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Preface 


The  purpose  of  this  report  is  to  increase  awareness  and  understanding  of  software  testing 
technologies  in  particular,  test  preparation,  test  execution,  test  evaluation,  and  source  code  static 
analysis  tools.  Use  of  this  report  should  be  the  first  step  in  transferring  effective  software  test 
processes,  methods,  and  tools  into  practical  use.  This  report  was  written  for  organizations 
responsible  for  the  development  and  maintenance  of  computer  software.  This  report  explains  how 
the  features  of  current  testing  technologies  can  improve  software  development  and  maintenance.  It 
includes  information  about  specific  products  in  the  marketplace.  The  information  is  aimed  at  those 
who  must  make  decisions  about  acquiring  advanced  technology  and  prepare  their  organizations  to 
use  it  effectively. 

This  August  1994  Software  Test  Technologies  Report  combines  the  February  1993  Test 
Preparation,  Execution,  and  Evaluation  [TPEE93]  Software  Technologies  Report  and  the  March 
1993  Source  Code  Static  Analysis  [SCSA93]  Technologies  Report  (Volumes  1  and  2)  and  updates 
those  versions  by  providing 

•  Updated  test  tool  taxonomy  and  test  technology  information. 

•  Guidance  for  improving  the  testing  process  using  the  Capability  Maturity  Model. 

•  Additional  testing  tools  and  updated  tool  information. 

•  Improved  Product  Sheet  format  that  condenses  relevant  information. 

•  Removal  of  redundant  information  between  the  TPEE  and  SCSA  reports. 

•  Updated  information  about  STSC  products  and  services. 

•  Updated  list  of  conferences  and  seminars. 

Note  that  the  Product  Critiques  and  tool  evaluation  characteristics  (criteria)  have  been 
removed  from  this  technology  report.  Current  Product  Critiques  and  evaluation  characteristics  can 
be  requested  from  the  STSC  at  any  time.  This  will  permit  better  STSC  service  and  support  in  the 
proper  use  of  the  information. 


(This  page  has  been  intentionally  left  blank.) 
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1  Introduction 

This  report  was  written  to  help  software  development  and  support  activities  (SDSA)*  identify, 
evaluate,  and  adopt  effective  software  testing  practices  and  technologies.  There  are  several  types  of 
software  testing  technologies  that  can  potentially  improve  an  organization's  ability  to  test  software 
to  find  defects  and  to  determine  if  the  software  satisfies  its  requirements.  Section  1  introduces  the 
Software  Technology  Support  Center  (STSC)  and  its  mission.  Section  2  describes  some  software 
testing  terms  and  a  classification  of  tools  (taxonomy).  Section  3  provides  additional  information 
about  static  analysis  technologies.  Section  4  presents  the  STSC's  method  for  evaluating  and  selecting 
software  test  tools.  Section  5  discusses  improving  the  testing  process  using  the  Software  Engineering 
Institute's  (SEI)  Capability  Maturity  Model  (CMM). 

1.1  The  Software  Technology  Support  Center 

The  STSC's  mission  is  to  assist  Air  Force  software  organizations  with  identifying,  evaluating, 
and  adopting  technologies  that  will  improve 

•  The  quality  of  their  software  products. 

•  Their  efficiency  in  producing  software. 

•  Their  ability  to  accurately  predict  the  cost  and  schedule  of  software  delivery. 

A  planned  approach  is  necessary  for  successful  transition.  In  general,  transitioning  effective 
practices,  processes,  and  technologies  consist  of  a  series  of  activities  or  events  that  occur  between 
the  time  a  person  encounters  a  new  idea  and  the  daily  use  of  that  idea.  Conner  and  Patterson's 
Adoption  Curve  [CONN82],  shown  in  Figure  1-1,  illustrates  these  activities.  After  encountering  a 
new  process  or  technology,  potential  customers  of  that  technology  increase  their  awareness  of  its 
usage,  maturity,  and  application.  If  the  process  or  technology  is  promising,  customers  try  to  better 
understand  its  strengths,  weaknesses,  costs,  and  applications.  These  first  activities  in  the  Adoption 
Curve  take  a  significant  amount  of  time. 

Promising  processes  and  technologies  are  then  evaluated  and  compared.  To  reduce  risk, 
customers  usually  try  new  processes  or  technologies  on  a  limited  scale  through  beta  tests,  case 
studies,  or  pilot  projects.  A  customer  then  adopts  processes  or  technologies  that  prove  effective. 


Software  Development  and  Support  Activity  is  a  DoD  or  military  service  organization  responsible  for  the  software 
development  or  support  of  a  designated  Mission-Critical  Computer  Resource  (MCCR).  Adaptation  is  based  on  [MCCR90]. 
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Finally,  refined  processes  and  technologies  become  essential  parts  of  an  organization's  daily  process 
(institutionalization). 


Figure  1-1.  Adoption  Curve. 


Word  processors  are  essential  in  most  organization's  daily  operations.  Yet,  30  years  ago  they 
did  not  exist.  The  institutionalization  of  word  processors  in  many  organizations  followed  a  series  of 
events  similar  to  those  identified  in  the  Adoption  Curve. 

The  STSC  is  researching  and  collecting  information  about  technologies  that  will  reduce  the 
time  and  resources  it  takes  to  become  aware,  understand,  evaluate,  select,  try,  and  adopt  effective 
practices,  processes,  and  technologies.  The  STSC  has  developed  the  following  objectives  to 
accomplish  its  mission: 

•  Information  Exchange 

Facilitate  the  exchange  of  better  software  business  practices,  processes,  and 
technologies  within  the  DoD. 

•  Technology  Evaluation 

Identify,  classify,  and  evaluate  effective  processes  and  technologies. 

•  Insertion  Projects 

Analyze  and  improve  processes,  support  the  adoption  of  new  methods  as 
needed,  evaluate  and  recommend  for  selection  effective  tools,  receive  and 
provide  appropriate  levels  of  training,  and  support  pilot  projects  to  try  out  and 
confirm  the  technology  insertion  efforts. 
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•  Technology  Champions 

Develop  technology  champions  who  can  infuse  effective  process  and 
technology  improvements  through  the  use  of  available  STSC,  DoD,  and 
industry  products,  services,  and  processes. 


1.2  STSC  Technology  Transition  Approach 

This  section  describes  the  STSC's  approach  to  meeting  the  objectives  identified  in  the  previous 

section. 


1.2.1  Information  Exchange 

This  objective  involves  exposing  potential  STSC  customers  to  available  technologies  and, 
conversely,  customer  requirements  to  technology  developers.  Referring  to  the  Adoption  Curve,  this 
objective  focuses  on  contact,  awareness,  and  understanding.  STSC  products  and  services  that 
accomplish  this  objective  include  CrossTalk  (the  DoD  journal  of  defense  software  engineering),  the 
annual  Software  Technology  Conference,  specific  technology  reports,  electronic  customer  services, 
and  introductory  workshops. 

1.2. 1.1  Crosstalk 

Over  13,000  software  professionals  receive  monthly.  This  publication  provides 

a  forum  for  the  exchange  of  ideas.  Articles  cover  leading  edge,  state-of-the-art,  and  state-of-the- 
practice  processes  and  technologies  in  software  engineering. 

1 .2. 1 .2  Software  Technology  Conference 

The  annual  Software  Technology  Conference  is  held  each  April  in  Salt  Lake  City,  Utah.  This 
conference  brings  together  over  2,400  software  professionals  from  government,  industry,  and 
academia  to  share  technology  solutions  and  exchange  ideas  and  information. 

1 .2. 1 .3  Technology  Reports 

STSC  technology  reports  provide  awareness  and  understanding  of  each  topic  in  preparation 
for  evaluation  and  selection  of  corresponding  technologies.  Over  34,000  of  these  reports  have  been 
distributed.  The  current  list  of  reports  include  the  following: 
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•  Software  Test  Technologies  Report  (this  report)  . 

•  Configuration  Management  [CM94]. 

•  Documentation  [DOC94]. 

•  Project  Management  [PM93]. 

•  Reengineering  [RE93]. 

•  Requirements  Analysis  and  Design  |RAD94]. 

•  Reuse  [REUS94]. 

•  Software  Engineering  Environments  [SEE94]. 

•  Software  Estimation  [EST93]. 

•  Software  Management  Guide  [SMG93]. 

The  STSC  also  provides  a  Metrics  Starter  Kit  |MET94]  that  helps  organizations  adopt  a 
measurement  program  that  satisfies  the  Air  Force  Software  Metrics  Policy,  93M-017  [DRUY94]. 

1 .2. 1 .4  Electronic  Customer  Services 

Along  with  the  services  mentioned  above,  the  STSC  also  provides  customers  with  access  to 
information  via  Electronic  Customer  Services  (ECS).  ECS  includes  a  bulletin  board  system,  which 
is  available  to  obtain  additional  information,  leave  messages,  add  information,  and  confer 
electronically.  ECS  can  be  accessed  via  the  INTERNET  at  address  137.241.33.1  or  stscbbs.af  mil 
or  by  calling  801-774-6509  with  modem  at  2400  or  9600  baud,  8  bit  word,  1  stop  bit,  and  no  parity. 

1 .2. 1 . 5  STSC  Introductory  Workshops 

Initial  visits  to  potential  STSC  customers  (see  Section  1.2.3)  can  be  arranged  to  introduce 
STSC  products  and  services  to  Air  Force  and  other  government  organizations.  The  STSC 
introductory  workshops  are  customized  to  introduce  concepts  about  specific  technologies  such  as 
testing,  metrics,  and  program  management.  An  STSC  introductory  workshop  provides  valuable  input 
to  begin  a  technology  insertion  project. 

1.2.2  Technology  Evaluation 

The  objective  of  technology  evaluation  involves  identifying  and  classifying  processes, 
methods,  and  technologies  that  can  potentially  improve  the  quality  or  productivity  of  software 
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development  and  maintenance.  Many  organizations  are  so  focused  on  deadlines  and  customer  needs 
that  they  lack  the  resources  and  time  to  thoroughly  investigate  options  for  improvement,  leaving  them 
vulnerable  to  the  marketing  hype  of  software  vendors.  The  STSC  has  developed  the  infrastructure 
to  provide  information  on  several  types  of  software  development  and  maintenance  technologies. 
Product  critiques,  essentially,  brief  tool  evaluations  from  experienced  technology  users,  are  collected 
and  distributed.  Quantitative  evaluations  are  detailed  and  objective  evaluations  performed  as  needed 
by  experienced  users  and  potential  users.  Quantitative  evaluations  are  performed  on  the  most 
promising  tools,  methods,  or  processes  using  the  STSC's  evaluation  guidelines  and  practices. 

1.2.3  Technology  Insertion  Projects 

STSC  technology  insertion  projects  are  customer-oriented  projects  that  assist  customers  in 
evaluating,  selecting,  and  piloting  new  processes,  methods,  and  technologies.  These  projects  include 
process  definition,  process  improvement,  methodology  insertion,  and  tool  insertion  and  are  often 
initiated  through  introductory  and  customized  workshops  addressing  specific  technology  areas. 

Referring  to  the  Adoption  Curve  (Figure  1-1),  an  insertion  project  helps  cement  understanding 
of  a  process  or  technology,  tailors  an  evaluation  of  the  process  or  technology  for  the  customer,  and 
pilots  the  use  of  that  process  or  technology  with  appropriate  levels  of  training.  Customers  move 
closer  to  adoption  of  the  process  or  technology  through  hands-on  experience.  It  is  important  to  try 
out  technology  improvements  in  a  pilot  project  to  confirm  that  the  technology  is  appropriate  for  the 
organization  and  that  the  organization  is  ready  and  able  to  adopt  the  new  technology. 

1.2.4  Technology  Champions 

Fowler  and  Przybylinski  [FOWL88]  propose  that  transitioning  new  technologies  from  a 
developer  to  a  consumer  requires  an  advocate  to  push  the  technology  and  a  receptor  to  pull  the 
technology  into  an  organization.  This  concept  is  illustrated  in  Figure  1-2. 

Effective  change  comes  from  within  the  organization.  The  objective  of  fostering  technology 
champions  is  to  develop  technology  receptors  within  individual  Air  Force  SDSAs.  These  receptors, 
technology  champions,  are  trained  in  the  use  of  the  STSC's  information,  products,  and  services  to 
enhance  their  organization's  ability  to  incorporate  advanced  practices,  processes,  and  technologies. 

Referring  to  the  Adoption  Curve  (Figure  1-1),  technology  champions  complete  the  trek  to 
institutionalization.  Champions  that  come  from  within  the  organization  should  be  politically  astute 
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and  aware  of  internal  organizational  requirements.  They  have  the  highest  probability  of  influencing 
the  adoption  and  daily  use  of  effective  business  practices,  processes,  and  technologies. 


The  STSC  helps  organizations  manage  their  technological  change  processes  by  helping  them 
define  their  strategic  and  tactical  plans.  Issues  such  as  management  and  staff  resistance  during  all 
improvement  phases  are  addressed  using  resources  from  the  Software  Engineering  Institute,  etc. 
Planning  for  and  confronting  resistance  in  an  atmosphere  of  understanding  and  support  will  help 
organizations  institutionalize  effective  processes  and  tools. 
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2  Introduction  to  Software  Testing  Technologies 

The  STSC's  Test  Technologies  Group  has  been  researching  software  testing  technologies 
including  practices  and  tools.  The  tools  include  government-owned  tools  as  well  as  commercial-off- 
the-shelf  (COTS)  tools.  This  report  identifies  a  number  of  tools  that  support  software  testing  in 
typical  development  and  maintenance  organizations  of  the  government.  Section  2. 1  defines  important 
testing  terms.  Section  2.2  characterizes  current  practices  in  software  testing.  Section  2.3  provides 
a  test  tool  classification  scheme. 

2.1  Testing  Terminology 

Definitions  of  some  commonly  used  testing  terms  will  help  introduce  state-of-the-practice 
software  testing  technologies.  To  start  with.  Bill  Hetzel's  definition  of  testing  is  provided  as  follows; 

"Testing  is  any  activity  aimed  at  evaluating  an  attribute  or  capability  of  a  program  or 

system  and  determining  that  it  meets  its  required  results"  [HETZ88]. 

This  definition  advances  the  concept  of  testing  as  a  process  of  executing  a  program  or  system 
with  the  intent  of  finding  errors  [MYER79].  Hetzel  suggests  that  there  are  many  ways  to  evaluate 
(or  test)  a  system  without  executing  it.  For  example,  you  can  test  a  requirements  specification  or 
design  document  by  building  test  cases  based  on  those  specifications  or  documents.  This  activity 
involves  testers  early  on  the  project  and  helps  correct  requirements  and  design  problems  before  they 
are  coded  when  they  are  more  expensive  to  fix.  Another  point  that  Hetzel  makes  is  that  our  intuitive 
understanding  of  testing  is  built  on  the  notion  of  "measuring"  or  "evaluating,"  not  trying  to  find 
errors.  He  says,  for  example,  that  we  don't  test  students  to  find  out  what  they  don't  know,  but  rather 
to  allow  them  to  demonstrate  an  acceptable  understanding  and  grasp  of  the  subject  matter. 

Both  views  of  testing  (any  evaluation  activity  and  executing  code  to  find  errors)  are 
important.  In  this  document,  testing  refers  to  the  act  of  detecting  the  presence  of  faults  in  code  or 
supporting  documentation,  or  demonstrating  their  absence  by  confirming  that  requirements  are  met, 
and  is  distinguished  fi'om  debugging  where  faults  are  isolated  and  corrected.  This  definition  of 
testing,  and  the  following  five  paragraphs,  define  several  important  testing  terms  and  were  adapted 
from  a  paper  entitled,  "An  Examination  of  Selected  Commercial  Software  Testing  Tools"  [IDA92]. 

An  error  is  a  mistake  made  by  a  software  developer.  Its  manifestation  may  be  a  textual 
problem  in  the  code  or  documentation  called  a  fault  or  defect.  A failure  occurs  when  an  encountered 
fault  prevents  software  from  performing  a  required  function  within  specified  limits. 
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Four  test  execution  stages  are  commonly  recognized:  unit  testing,  integration  testing,  system 
testings  and  acceptance  testing.  In  unit  testing,  each  program  module  is  tested  in  isolation,  often  by 
the  developer.  In  integration  testing,  these  modules  are  combined  so  that  successively  larger  groups 
of  integrated  software  and  hardware  modules  can  be  tested.  System  testing  examines  an  integrated 
hardware  and  software  system  to  verify  that  the  system  meets  its  specified  requirements.  Acceptance 
testing  is  generally  a  select  subset  of  system  test  cases  that  formally  demonstrates  key  functionality 
for  final  approval  and  is  usually  performed  after  the  system  is  installed  at  the  user's  site. 

In  bottom-up  testing,  the  modules  at  the  bottom  of  the  invocation  hierarchy  are  tested 
independently  using  test  drivers,  then  modules  at  the  next  higher  level  that  call  these  modules  are 
integrated  and  tested,  and  so  on.  Top-down  testing  starts  at  the  highest-level  module,  with  stubs 
replacing  the  modules  it  invokes.  These  stubs  are  then  replaced  by  the  next  lower-level  modules,  with 
new  stubs  being  provided  for  the  modules  that  these  call,  and  so  on. 

Dynamic  analysis  approaches  rely  on  executing  a  piece  of  software  to  determine  if  the 
software  functions  as  expected.  This  can  involve  running  the  software  in  a  special  test  environment 
with  stubs,  drivers,  simulators,  test  data,  and  other  special  conditions  or  running  the  software  in  an 
actual  operating  environment  with  real  data  and  real  operating  conditions.  The  effectiveness  of  any 
d3mamic  analysis  technique  is  directly  related  to  the  test  data  used.  Current  tools  attempt  to  detect 
faults  rather  than  demonstrate  the  absence  of  faults.  Additionally,  most  of  these  tools  can  only  detect 
faults  whose  effects  propagate  to  software  outputs,  unless  the  software  has  been  specially 
instrumented  to  monitor  internal  data  elements  {intrusive)  or  special  hardware  monitors  have  been 
attached  to  the  system  (nonintrusive). 

Static  analysis  refers  to  the  evaluation  of  software  without  executing  it  using  automated  (tool 
assisted)  and  manual  mechanisms  such  as  desk-checking,  inspections,  reviews,  and  walkthroughs. 
Static  analyzer  tools  can  demonstrate  the  absence  of  certain  types  of  defects  such  as  variable  typing 
errors.  Static  analysis  alone  cannot  detect  faults  that  depend  on  the  underl5dng  operating 
environment.  Consequently,  effective  testing  requires  a  combination  of  static  and  dynamic  analysis 
approaches. 

In  support  of  dynamic  analysis,  different  strategies  or  heuristics  can  be  used  to  drive  test  data 
generation.  Commercial  automated  support  is  currently  available  for  both  functional  and  structural 
strategies.  Functional  {black  box)  tests  are  derived  from  system-level,  interface,  and  unit-level 
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specifications.  Structural  (y^hite  box)  tests  require  knowledge  of  the  source  code  including  program 
structure,  variables,  or  both.  With  functional  strategies,  test  data  is  derived  from  the  program's 
requirements  with  no  regard  to  program  structure.  Functional  approaches  are  language-independent. 
In  structural  strategies,  test  data  is  derived  from  the  program's  structure. 

Functional  strategies  can  be  applied  at  all  testing  levels.  System  tests  can  be  defined  at  the 
requirements  analysis  phase  to  test  overall  software  requirements.  During  the  design  phase, 
integration  tests  can  be  defined  to  test  design  requirements.  During  the  coding  phase,  unit  tests  can 
be  defined  to  test  coding  requirements. 


Figure  2.1  illustrates  the  software  development  lifecycle  with  test  development  and  execution 
activities.  The  left  side  of  the  figure  identifies  the  specification,  design,  and  coding  activities  for 
developing  software.  It  also  indicates  when  the  test  specification  and  test  design  activities  can  start. 
For  example,  the  system/acceptance  tests  can  be  specified  and  designed  as  soon  as  software 
requirements  are  known.  The  integration  tests  can  be  specified  and  designed  as  soon  as  the  software 
design  structures  are  known.  And  the  unit  tests  can  be  specified  and  designed  as  soon  as  the  code 
units  are  prepared. 


Specii^^/Dcsign  Code 

Syatem/Acceptance  Testa _ 


Execute 

Design 

Integration 

Tests 

ISpecify/Pesigii  Code 

_ Integration  Teats _ 


Code 


Execute 

Unit 

Tests 


I  Specify/Design  Code 
Unit  TcBte 


Figure  2-1.  Software  Development  Lifecycle  -  Modified  V  Model. 
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Section  5.3.2. 1  suggests  that  building  tests  may  be  the  most  objective  type  of  testing  for 
requirements  and  design  specifications  before  code  is  available  to  execute.  This  may  also  be  some 
of  the  least  expensive  testing  that  we  can  do.  The  right  side  of  the  figure  identifies  when  the 
evaluation  activities  occur  that  are  involved  with  executing  and  testing  the  code  at  its  various  stages 
of  evolution. 

Requirements-hased  test  case  generators  create  fiinctional  test  cases  by  random,  algorithmic, 
or  heuristic  means  that  can  be  applied  at  all  functional  test  levels.  Superior  quality  test  case 
generators  vwll  use  all  three  means.  In  random  or  statistical  test  case  generation,  the  tool  chooses 
input  structures  and  values  to  form  a  statistically  random  distribution.  In  algorithmic  test  case 
generation,  the  tool  follows  a  set  of  rules  or  procedures.  Several  popular  algorithms  or  methods 
employed  by  test  case  generators  include  equivalence  class  partitioning,  boundary  value  analysis, 
and  cause-effect  graphing.  When  generating  test  cases  by  heuristic  or  failure-directed  means,  the 
tool  uses  information  fi’om  the  tester.  Failures  that  the  tester  discovered  in  the  past  are  entered  into 
the  tool.  The  tool  uses  that  history  of  failures  to  generate  test  cases  [POST92]. 

Equivalence  class  partitioning  involves  identifying  a  finite  set  of  representative  input  values 
that  invoke  as  many  different  input  conditions  as  possible  [MYER79].  Test  cases  that  exercise 
boundary  conditions  usually  have  a  higher  payoff  than  other  t5q5es  of  test  cases  [MYER79].  Building 
test  cases  to  exercise  specific  functions  supports  demonstration  of  requirements.  Regarding  cause- 
effect  graphing,  causes  relate  to  distinct  input  conditions,  and  effects  relate  to  the  resulting  output 
conditions  or  system  transformations.  Test  data  is  derived  from  a  combinational  logic  network  that 
represents  the  logical  relationships  between  causes  and  effects. 

The  structural  strategies  at  the  unit  testing  level  include  statement  coverage,  branch  coverage, 
and  path  coverage  testing.  Statement  coverage  only  detects  which  statements  have  been  executed 
and  is  not  considered  adequate  for  structural  testing.  Branch  coverage  testing  requires  each 
conditional  branch  statement  and  the  code  segment  whose  execution  is  controlled  by  this  conditional 
to  be  executed  at  least  once.  A  variation  of  branch  coverage  involves  exercising  all  feasible  true  and 
false  outcomes  of  each  logical  component  of  compounded  conditional  statements.  Path  coverage 
testing  requires  execution  of  every  path  including  loops.  Path  testing  is  the  more  stringent  strategy 
but  can  incur  unacceptable  computational  costs.  There  are  variations  of  path  coverage  testing  that 
require  basis  paths  be  executed  (see  [MCCA89]  for  a  complete  discussion  of  basis  paths)  or  that 
require  at  least  one  loop  iteration  be  executed  for  all  loops  in  addition  to  all  conditionals.  A 
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structural  coverage  strategy  at  the  integration  level  requires  that  each  pair  of  module  invocations  be 
executed  at  least  once. 

Prototyping  is  becoming  more  widely  accepted  and  implemented  as  an  iterative  development 
activity  on  many  projects.  The  use  of  this  technique  is  being  accelerated  by  the  availability  of  more 
automated  tools  that  enable  quicker  and  easier  prototyping  of  system  components.  Prototyping 
evaluates  (tests)  requirements  specifications  at  the  conceptualization  phase  or  the  requirements 
analysis  phase  and  can  save  a  considerable  amount  of  development  time  when  properly  managed. 

Verification  is  defined  by  the  MIL-STD-2167A  as  "the  process  of  evaluating  the  products  of 
a  given  sofl;ware  development  activity  to  determine  correctness  and  consistency  with  respect  to  the 
products  and  standards  provided  as  input  to  that  activity"  [216788].  Verification  is  a  testing  activity 
that  occurs  at  all  lifecycle  phases  using  the  inputs  of  a  phase  to  evaluate  the  products  of  that  phase. 
Validation  is  defined  by  the  MIL-STD-2167A  as:  "the  process  of  evaluating  software  to  determine 
compliance  with  specified  requirements"  [216788].  Validation  ensures  that  software  meets  the 
requirements  as  specified  at  all  requirements  definition  phases,  i.e.,  phases  that  define  system 
requirements,  software  requirements,  design  requirements,  and  coding  requirements. 

2.2  Software  Testing:  Current  Practices 

Many  organizations  approach  software  testing  with  the  same  practices  and  tools  they  used 
10  or  more  years  ago.  This  was  evident  in  a  survey  conducted  by  Software  Quality  Engineering,  Inc. 
of  leading  software  organizations  [SOFT90].  While  many  recommended  testing  practices  are  more 
than  10  years  old,  there  is  more  tool  support  available  for  testing  software  in  today's  market. 
Organizations  should  regularly  review  their  testing  practices  and  industry  practices  for  potential 
opportunities  for  improvement. 

Test  development,  execution,  and  analysis  is  still  labor-intensive  like  most  development 
activities.  Testing  can  consume  over  50  percent  of  software  development  costs  (note  that  testing 
costs  should  not  include  debugging  and  rework  costs).  In  one  particular  case,  NASA's  Apollo 
program,  80  percent  of  the  total  software  development  effort  was  incurred  by  testing  [DUNN84]. 
In  general,  schedule  pressure  limits  the  amount  of  testing  that  can  be  performed.  Furthermore, 
defects  fi'equently  lead  to  failure  of  operational  software.  Barry  Boehm  tells  us  that  3  to  10  failures 
per  thousand  lines  of  code  (KLOC)  are  typical  for  commercial  software,  and  1  to  3  failures  per  KLOC 
are  typical  for  industrial  software  [BOEH88].  With  a  rate  of  0.01  failures  per  KLOC  for  its  shuttle 
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code,  however,  NASA  has  demonstrated  that  lower  defect  counts  can  be  achieved  [HEND94].  The 
cost  of  correcting  defects  increases  as  software  development  progresses  for  example,  the  cost  of 
fixing  a  requirements  fault  during  operation  can  be  60  to  100  times  the  cost  of  fixing  that  same  fault 
during  early  development  stages  [PRES87].  Consequently,  timely  defect  detection  is  important. 

Improved  practices  and  automated  tools  can  reduce  testing  costs.  In  addition  to  eliminating 
some  repetitive  manual  tasks,  tools  can  promote  effective  dynamic  analysis  by  guiding  the  selection 
of  test  data  and  monitoring  test  executions.  Through  capturing  and  reporting  data  gathered  during 
the  performance  of  testing  activities,  tools  also  support  quantitative  process  measurement  that  is 
necessary  for  controlling  the  testing  process.  Benefits  claimed  by  some  of  the  tools  discussed  later 
include^ 

•  A  coverage  analyzer  tool  that  has  saved  a  developer  $1 5,000  or  more  per  KLOC. 

•  A  requirements-based  test  case  generator  that  has  given  clients  an  8;1  reduction  in 
test  development  effort,  and  one  client  has  achieved  a  reduction  from  1.3  to  0.072 
failures  per  KLOC. 

Of  course,  testing  tools  are  not  the  only  mechanism  for  improving  software  quality,  reliability, 
and  productivity.  Software  inspections,  for  example,  have  been  reported  to  find  60  to  90  percent  of 
software  defects,  while  reducing  total  development  costs  by  as  much  as  25  percent  [FAGA86]. 

2.3  Software  Test  Tool  Classification 

The  STSC  has  been  collecting  information  from  several  sources  on  classifying  test  tools.  This 
information  is  being  used  to  help  identify,  evaluate,  and  select  software  test  tools.  The  following 
books  or  articles  have  contributed  to  STSC's  Test  Tool  Classification  Scheme: 

•  Evaluation  and  Validation  (E&V)  Reference  Manual  [CLAR91]. 

•  A  Complete  Guide  to  Software  Testing  [HETZ88]. 

•  Testing  Tools  Reference  Guide  [DURA93]. 

•  CAST  Report  [GRAH93]. 

•  A  Complete  Toolkit  for  the  Software  Tester  [POST92]. 


^ool  names  are  omitted  since  these  claims  have  been  neither  validated  nor  invalidated. 
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•  Software  Testing  and  Evaluation  [DEMI87]. 

•  An  Examination  of  Selected  Commercial  Software  Testing  Tools;  1992  [IDA92]. 

The  STSC  Test  Tool  Classification  Scheme  is  primarily  based  on  the  E&V Reference  Manual, 
which  contains  a  wealth  of  information  about  development  functions,  maintenance  functions,  and 
quality  attributes  for  evaluating  Ada  Programming  Support  Environments  (APSEs).  This  information 
is  relevant  to  many  development  environments  besides  Ada  and  identifies  lifecycle  functions  required 
for  development  and  maintenance  using  the  MIL-STD-2167A. 

Some  popular  terms  for  software  development  and  maintenance  tools  such  as 
Computer-Aided  Software  Engineering  (CASE)  and  Computer-Aided  Software  Testing  (CAST)  have 
evolved  over  the  last  few  years.  Unfortunately,  these  terms  have  caused  some  confusion  because  they 
can  be  interpreted  to  mean  that  any  software  tool  that  supports  a  software  engineering  function,  such 
as  a  compiler,  is  a  CASE  tool.  Equally,  any  software  tool  that  supports  testing,  such  as  a  timing 
analyzer,  can  be  considered  a  CAST  tool.  No  attempt  is  made  in  this  report  to  limit  the  scope  of 
tools  that  these  terms  address.  However,  as  the  industry  moves  toward  building  Software 
Engineering  Environments  (SEEs)  or  APSEs,  the  CASE  and  CAST  tools  of  particular  interest  and 
importance  are  those  that  can  integrate  development  and  testing  lifecycle  activities  or  perform 
multiple  related  functions  within  a  lifecycle  activity. 

Some  testing  tools  perform  more  than  one  testing  function  or  support  more  than  one  lifecycle 
activity.  Since  vendors  package  various  capabilities  within  their  tool  sets,  the  same  tool  can  appear 
under  several  tool  classifications.  Multiple  classifications  are  used  in  tool  catalogs  such  as  the  Testing 
Tools  Reference  Guide  [DURA93].  The  important  point  here  is  that  software  engineers  need  to  be 
able  to  find  tools  that  perform  the  required  functions. 

The  STSC  tries  to  find  tools  that  support  all  fimctions  in  all  lifecycle  phases  of  development 
or  maintenance  for  most  major  platforms.  However,  some  tools  may  provide  a  specific  capability, 
but  not  be  listed  under  that  corresponding  tool  type.  The  tool  should  be  recognized  by  the  industry 
as  that  specific  type  of  tool.  Classification  of  tools  can  be  very  difficult  given  the  variety  of  published 
test  tool  hierarchies,  the  number  and  "flavor"  of  tools  in  the  market,  and  the  terminology  used  to 
describe  them.  The  STSC  classification  scheme  is  not  intended  to  be  complete,  nor  is  it  intended  to 
conflict  with  other  schemes.  As  new  technologies  emerge  and  are  made  commercially  available,  new 
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types  will  be  added.  Lists  of  tool  characteristics  are  then  provided  to  help  customers  focus  their 
testing  requirements  and  help  them  find  the  right  tools. 

The  industry  has  accepted  the  notion  of  testing  at  all  lifecycle  phases.  The  testing  activities 
may  be  called  by  different  names  but  for  the  purposes  of  grasping  the  extent  of  the  role  of  testing 
throughout  the  software  lifecycle,  all  tools  that  perform  a  test  support  or  evaluation  role  are 
categorized  as  testing  tools.  For  example,  test  planning  is  an  essential  element  of  the  testing  role 
although  it  doesn't  directly  produce  test  results  in  and  of  itself  Testing  activities  can  be  initially 
classified  under  three  major  headings; 

•  Test  resource  management. 

•  Testing  at  requirements  analysis  and  design  phases. 

•  Testing  at  implementation  and  maintenance  phases. 

Figure  2-2  presents  the  STSC  Test  Tool  Classification  Scheme.  This  classification  scheme 
is  not  intended  to  dictate  how  organizations  should  classify  test  tools  but  rather  provides  a  starting 
point  from  which  to  discuss  automated  testing  capabilities.  Tools  that  manage  test  resources  and 
several  types  of  tools  that  support  testing  at  the  requirements  analysis  and  design  phases  are  discussed 
in  other  STSC  technology  reports  (references  are  provided  below). 

This  report  focuses  on  software  test  tools  that  support  the  implementation  and  maintenance 
phases  and,  more  particularly,  those  that  support  source  code  static  analysis,  test  preparation,  test 
execution,  and  test  evaluation  of  the  actual  software  system  at  the  unit,  integration,  system,  and 
acceptance  testing  levels.  Note  that  requirements-based  test  case  generators  and  test  planners  appear 
under  two  major  headings.  This  was  done  to  recognize  the  important  role  that  these  tools  play  during 
the  requirements  analysis  and  design  phases  and  during  the  implementation  and  maintenance  phases. 

2.3.1  Test  Resource  Management  Tools 

Management  of  testing  resources  is  required  at  all  lifecycle  phases.  The  following  paragraphs 
describe  several  types  of  test  resource  managers: 
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Test  Resource  Management  Tools  (Section  2.3.1) 

Configuration  Managers 
Project  Managers 

Requirements  and  Design  Test  Support  Tools  (Section  2.3.2) 

Analyzers  for  Software  Plans,  Requirements,  and  Designs 

System/Prototype  Simulators 

Requirements  Tracers 

Requirements-Based  Test  Case  Generators 

Test  Planners 

Implementation  and  Maintenance  Test  Support  Tools  (Section  2.3.3) 
Compilers 

Source  Code  Static  Analyzers 

Auditors 

Complexity  Measurers 
Cross  Referencing  Tools 
Size  Measurers 
Structure  Checkers 
Syntax  and  Semantics  Analyzers 

Test  Preparation  Tools 

Data  Extractors 

Requirements-Based  Test  Case  Generators 
Test  Data  Generators 
Test  Planners 

Test  Execution  Tools  (Dynamic  Analyzers) 

Assertion  Analyzers 
Capture-Replay  Tools 
Coverage/Frequency  Analyzers 
Debuggers 
Emulators 
Network  Analyzers 
Peiformance/Timing  Analyzers 
Run-Time  Error  Checkers 
Simulators 

Status  Displayer/Session  Documenters 
Test  Execution  Managers 
Validation  Suites 

Test  Evaluators 

Comparators 

Data  Reducers  and  Analyzers 

_ Defect/Change  Trackers _ _ 


Figure  2-2.  STSC  Test  Tool  Classification  Scheme. 

a.  Configuration  managers  monitor  and  control  the  effects  of  changes  throughout  development 
and  maintenance  and  preserve  the  integrity  of  released  and  developed  versions.  Change 
control  of  software  and  test  documentation  (including  test  plans,  test  requirements,  test 
procedures,  and  test  cases)  must  be  carefully  managed.  Configuration  managers  can  be  some 
of  your  most  powerful  testing  tools.  Code  version  retrieval,  defect  tracking,  change  request 
management,  and  code  change  monitoring  capabilities  are  typical  features  of  configuration 
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management  systems.  Configuration  management  technologies  are  analyzed  and  published 
in  the  Configuration  Management  Technologies  Report  [CM94]. 

b.  Project  managers  help  managers  plan  and  track  the  development  and  maintenance  of  systems. 
These  tools  document  the  estimates,  schedules,  resource  requirements,  and  progress  of  all 
project  activities.  Test  planning  activities  are  often  neglected,  delayed,  or  impacted  by  delays 
in  early  lifecycle  phases.  Project  managers  can  elevate  the  role  of  testing  in  a  project  with 
management's  documented  commitment  (see  Section  5,  Improving  the  Testing  Process  Using 
the  Capability  Maturity  Model,  for  more  information  on  this  subject).  Project  management 
technologies  are  analyzed  in  the  Project  Management  Technologies  Report  [PM93]. 

2.3.2  Requirements  and  Design  Test  Support  Tools 

It  is  widely  acknowledged  that  testing  must  be  considered  at  both  the  requirements  analysis 
and  design  phases.  Software  requirements  and  design  information  provide  primary  input  to  define 
test  requirements  and  prepare  the  test  plans.  CASE  tools  that  support  the  requirements  analysis  and 
design  phases  are  often  called  Upper-CASE  tools.  The  following  paragraphs  describe  several 
requirements  and  design  test  support  tools; 


a.  Analyzers  for  software  plans,  requirements,  and  designs  evaluate  the  specifications  for 
consistency,  completeness,  and  conformance  to  established  specification  standards.  These 
tools  are  reviewed  in  ihs  Requirements  Analysis  and  Design  Technologies  Report  [RAD94]. 

b.  System/Prototype  simulators  merge  analysis  and  design  activities  with  testing.  Requirements 
are  refined  while  obtaining  rapid  feedback  of  analysis  and  design  decisions.  Requirements  can 
be  initially  validated,  altered,  or  canceled  by  demonstrating  critical  system  fonctions  much 
quicker  than  is  possible  under  normal  full-scale  development. 

Prototyping  tools  allow  more  efiBcient  consideration  of  design  alternatives,  evaluation  of  user 
interfaces,  and  feasibility  testing  of  complex  algorithms.  Prototyping  must  be  carefiilly 
monitored  and  controlled  to  ensure  that  appropriate  full-scale-development  objectives  are 
considered.  Otherwise,  prototj^ing  can  result  in  ad  hoc  development  with  no  documentation. 
It  is  anticipated  that  prototype  simulators  will  be  reviewed,  and  a  report  will  be  written  as 
STSC  customer  interest  prescribes. 

c.  Requirements  tracers  can  significantly  reduce  the  work  effort  of  tracing  requirements  to 
associated  design  information,  source  code,  and  test  cases  for  large  projects.  These  tools 
provide  links  between  requirements  and  design,  code,  and  test  cases.  What  has  normally  been 
a  lengthy,  manual  process  can  be  significantly  automated  using  these  tools.  Requirements 
tracers  are  reviewed  in  the  Requirements  Analysis  and  Design  Technologies  Report 
[RAD94]. 

d.  Requirements-based  test  case  generators  can  help  developers  evaluate  requirements  and 
design  information  during  the  requirements  analysis,  design,  and  code  phases.  Early 
consideration  of  the  types  of  test  cases  that  a  tool  builds  encourages  developers  to  improve 
requirements  and  designs  to  pass  those  kinds  of  tests.  See  Section  2.3.3.b  for  more 
information. 
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e.  Test  planners  assist  developers  in  planning  and  defining  acceptance,  system,  integration,  and 

unit-level  tests.  Test  cases  (including  test  inputs,  expected  results,  and  test  procedures) 
should  be  defined  and  designed  early  in  the  development  lifecycle  to  help  prevent  errors.  See 
Section  2.3  3.b  for  more  information. 

2.3.3  Implementation  and  Maintenance  Test  Support  Tools 

Some  CASE  tool  vendors  advertise  full  lifecycle  support  because  they  provide  consistency 
and  completeness  checking  of  the  requirements  and  design  specifications,  some  of  which  offer  no 
testing  support  at  the  implementation  and  maintenance  phases.  This  has  increased  development  time 
because  unit,  integration,  and  system-level  testing  needs  were  not  properly  considered  at  the 
requirements  analysis  and  design  phases  [SIEG91].  Note  that  some  of  these  t5q)es  of  tools  are  built 
in-house  and  then  discarded  (or  adapted  for  other  projects)  after  use.  The  following  paragraphs 
describe  implementation  and  maintenance  test  support  tools: 


a.  Compilers  are  not  generally  considered  testing  tools,  even  though  testing  source  code  is  a 
major  function  of  these  tools.  It  is  anticipated  that  compilers  (particularly  Ada  compilers)  will 
be  reviewed,  and  a  report  will  be  written  as  STSC  customer  interest  prescribes. 

b.  Source  code  static  analyzers  examine  source  code  without  executing  it.  Static  analyzers 
extend  the  analysis  performed  by  compilers.  Various  kinds  of  static  analysis  tools  are 
available.  See  Section  3  for  more  information  on  static  analysis  technologies. 

(1)  Auditors  analyze  code  to  ensure  conformance  to  established  rules  and  standards. 
Typical  rules  and  practices  include  adherence  to  structured  design  and  coding 
constructs,  use  of  portable  language  subsets,  or  use  of  a  standard  coding  format 
[DEMI87]. 

(2)  Complexity  measurers  compute  metrics  from  the  source  code  to  determine  various 
complexity  attributes  associated  with  the  source  code  or  designs  written  in  a  program 
design  language  (PDL).  This  is  accomplished  by  evaluating  program  characteristics 
such  as  control  flow,  operands/operators,  data,  and  system  structure. 

(3)  Cross  referencing  tools  provide  referencing  between  various  entities.  Some  of  these 
tools  provide  a  comprehensive  on-line  cross-referencing  capability.  Some  of  the  types 
of  data  that  cross  referencing  tools  provide  include  cross  indexes  of  statement  label, 
data  name,  literal  usage,  and  intersubroutine  calls. 

(4)  Size  measurers  count  source  lines  of  code  (SLOC).  SLOC  counting  tools  typically 
provide  counts  for  comments,  executable  lines,  semicolons,  declarations,  total  lines, 
etc.  Some  of  these  tools  automatically  collect  code  information  and  provide  historical 
databases  that  track  code  growth,  changes,  and  trends. 

(5)  Structure  checkers  identify  some  structure  anomalies  and  portray  the  structure  of  the 
source  code  through  graphics  or  text.  Examples  of  typical  charts  that  are  produced 
are  di-graphs  (directed  graphs),  structure  charts,  data  flow  diagrams,  flowcharts,  and 
call  trees.  Note  that  the  static  data  flow  and  static  path  flow  analyzer  tool  types 
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identified  in  [TPEE93]  and  [SCSA93]  have  been  included  with  structure  checkers  to 
simplify  the  classification  scheme  in  this  report. 

(6)  Syntax  and  semantics  analyzers  have  been  traditionally  called  static  analyzers. 
Compilers  perform  these  functions  but  usually  with  limited  scope.  FORTRAN  S5mtax 
and  semantic  analyzers  are  often  needed  to  identify  type  conflicts  in  calling  arguments 
of  separately  compiled  subroutines.  Ada  compilers  do  not  have  this  problem  due  to 
the  characteristics  of  the  Ada  language.  Other  syntax  and  semantic  analyzers  (such 
as  UNIX  lint)  identify  unused  variables. 

Appendix  A.  3  contains  lists  of  source  code  static  analyzers. 

c.  Test  preparation  tools  include  tools  that  prepare  test  data  or  test  case  information  that  may 

require  various  levels  of  follow-on  formatting.  Also  included  with  the  test  preparation  tools 

are  the  test  planners. 

(1)  Data  extractors  build  test  data  from  existing  databases  or  test  sets.  Call  the  STSC  for 
a  current  list  of  data  extractors. 

(2)  Requirements-based  test  case  generators  help  developers  evaluate  code  requirements 
by  building  test  cases  from  requirements  written  following  the  rules  of  the  tool's 
formal  specification  language.  (See  Sections  2.1  and  2.3. 2.d  for  more  information.) 
Appendix  A.  3  contains  a  list  of  requirements-based  test  case  generators. 

(3)  Test  data  generators  build  test  inputs  that  are  formatted  (or  can  be  readily  formatted) 
in  the  required  files.  Some  test  data  generators  build  statistically  random  distributed 
test  data  sets.  (See  Section  2. 1  for  more  information.)  Call  the  STSC  for  a  current 
list  of  test  data  generators. 

(4)  Test  planners  assist  developers  in  planning  and  defining  tests.  (See  Section  2.3. 2.e 
for  more  information.)  Call  the  STSC  for  a  current  list  of  test  planners. 

d.  Test  execution  tools  dynamically  analyze  the  software  to  be  tested. 

(1)  Assertion  analyzers  instrument  the  code  with  logical  expressions  that  specify 
conditions  or  relations  among  the  program  variables.  Call  the  STSC  for  a  current  list 
of  assertion  analyzers. 

(2)  Capture-replay  tools  automatically  record  test  inputs  (capture  scripts)  and  replay 
those  test  inputs  (playback  scripts)  in  subsequent  tests  after  code  changes.  These 
tools  can  dramatically  improve  tester  productivity.  Some  of  these  tools  can  fiilly 
automate  regression  testing  when  combined  with  the  capability  to  automatically 
compare  previous  results  with  current  outputs.  Many  communications  programs 
(e.g.,  PROCOMM  PLUS)  provide  scripting  capabilities  that  could  support  some 
testing  activities  such  as  capture  of  test  sequences.  (These  tools  were  not  included 
because  testing  was  not  an  advertised  function.)  Appendix  A.3  contains  a  list  of 
capture-replay  tools. 

(3)  Coverage/frequency  analyzers  assess  the  coverage  of  test  cases  with  respect  to 
executed  statements,  branches,  paths  or  modules.  These  tools  generally  require 
instrumentation  of  the  code  to  be  able  to  monitor  coverage;  that  is,  special  code  is 
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added  to  monitor  the  execution  paths.  (See  Section  2.1  for  more  information.) 
Appendix  A.3  contains  a  list  of  coverage/frequency  analyzers. 

(4)  Debuggers  often  directly  support  the  testing  effort  even  though  their  prime  intent  is 
to  locate  errors  resulting  from  testing.  Some  debuggers  have  coverage  analysis 
capabilities  that  directly  support  testing.  Debuggers  can  be  used  to  perform  various 
low-level  testing  fimctions  and  are  the  only  test  execution  tools  that  some 
organizations  have.  Contact  the  STSC  for  a  current  list  of  debuggers. 

(5)  Emulators  may  be  used  in  place  of  missing  or  unavailable  system  components. 
Emulators  are  generally  hardware  simulations  of  various  system  components  that 
usually  operate  at  the  real-time  speed  of  the  components  being  emulated.  Terminal 
emulation  is  a  capability  commonly  found  in  communications  tools,  some  of  which 
may  be  used  for  testing  software.  Emulation  is  usually  done  for  economic  or  safety 
reasons.  Either  the  emulated  components  are  not  yet  available  or  they  are  too 
expensive  to  waste  by  permitting  some  destructive  functions  to  be  tested.  Call  the 
STSC  for  a  current  list  of  emulators. 

(6)  Network  analyzers  are  a  special  class  of  testing  tools  that  draw  upon  the  technologies 
of  several  other  types  of  tools.  These  tools  have  the  capability  to  analyze  the  traffic 
on  the  network  to  identify  problem  areas  and  conditions.  These  analyzers  often  allow 
you  to  simulate  the  activities  of  multiple  terminals.  Call  the  STSC  for  a  current  list 
of  network  analyzers. 

(7)  Peiformance/timing  analyzers  monitor  timing  characteristics  of  software  components 
or  entire  systems.  Appendix  A.3  contains  a  list  of  performance/timing  analyzers. 

(8)  Eun-time  error  checkers  monitor  programs  for  memory  referencing,  memory  leaking 
(using  memory  outside  program  space),  or  memory  allocation  errors.  These  tools 
may  also  automatically  monitor  the  use  of  stacks  and  queues.  Appendix  A.3  contains 
a  list  of  run-time  error  checkers. 

(9)  Simulators  are  used  in  place  of  missing  or  unavailable  system  components. 
Simulators  are  generally  software  implementations  of  hardware  components  in  which 
only  the  necessary  characteristics  are  simulated  in  software.  Example  types  of 
simulators  include  environmental,  functional,  and  instruction  simulators.  Like 
emulation,  simulation  is  also  done  for  economic  or  safety  reasons.  Call  the  STSC  for 
a  current  list  of  simulators. 

(10)  Status  displayers/session  documenters  provide  test  status  information  and  record 
selected  information  about  a  test  run.  Call  the  STSC  for  a  current  list  of  status 
displayers/session  documenters. 

(11)  Test  execution  managers  are  a  general  classification  of  test  tools  that  automate 
various  functions  of  setting  up  test  runs,  performing  a  variety  of  tests,  and  cleaning 
up  after  a  test  to  reset  the  system.  Test  execution  managers  perform  many  of  the 
same  fianctions  as  capture-replay  tools  by  automatically  executing  test  cases  using 
scripts  and  sets  of  input  files.  Test  execution  managers  additionally  maintain  a  test 
results  history.  This  category  includes  tools  that  have  been  termed  test  drivers,  test 
harnesses,  and  test  executives.  Appendix  A.3  contains  a  list  of  test  execution 
managers. 
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(12)  Validation  suites  validate  software  against  a  well-defined  standard  such  as  the  Ada 
coding  standard  ANSI/MIL-STD-1815A,  12  Jan  83,  and  are  often  used  with 
compilers  and  operating  systems.  Call  the  STSC  for  a  current  list  of  validation  suites. 

e.  Test  evaluators  include  a  variety  of  off-the-shelf  and  system  specific  tools  that  perform  time- 

consuming,  error-prone,  and  boring  functions. 

(1)  Comparators  compare  entities  with  each  other  after  a  software  test  and  note  the 
differences.  Capture-replay  tools  often  provide  a  dynamic  comparison  capability, 
which  compares  entities  while  the  software  is  under  test.  Appendix  A.3  contains  a  list 
of  comparators. 

(2)  Data  reducers  and  analyzers  convert  data  to  a  form  that  can  be  more  readily 
interpreted  and  can  sometimes  perform  various  statistical  analyzes  on  the  data.  Key 
information  can  be  extracted  from  execution  logs  to  determine  correctness  or 
appropriateness  of  system  behavior  [CLAR91].  Call  the  STSC  for  a  current  list  of 
data  reducers  and  analyzers. 

(3)  Defect/Change  Trackers  keep  track  of  error  information  and  generate  error  reports. 
Accounting  for  requirements  and  design  errors  should  be  considered  in  these  tools, 
defect/change  trackers  are  often  part  of  configuration  management  systems,  (see 
Section  2.3. l.a).  Also,  they  can  be  integrated  in  Software  Engineering  Environments 
to  automatically  keep  track  of  error  information,  saving  engineers  and  managers 
considerable  error  reporting  time.  Refer  to  the  Software  Engineering  Environments 
Report  [SEE94]  for  more  iirformation  about  integrating  defect  tracking  into  an  SEE. 
Appendix  A.3  contains  a  list  of  defect/change  trackers. 
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3  Static  Analysis  Technologies 

A  group  of  testers  from  the  Institute  of  Defense  Analysis  identified  several  static  analysis 
techniques  in  their  paper,  SDS  Software  Testing  and  Evaluation:  A  Review  of  the  State  of  the  Art  in 
Software  Testing  and  Evaluation  with  Recommended  R&D  Tasks  [YOUN89].  These  techniques 
complement  the  static  analysis  functions  listed  in  the  ESiV Reference  Manual.  Christine  Youngblut 
categorized  the  static  analysis  techniques  into  four  groups: 

"The  first  group  consists  of  those  techniques  which  produce  general  information 
about  a  program,  for  example,  symbol  cross-referencers,  rather  than  search  for  actual 
faults.  These  are  relatively  common  and  often  provided  by  a  compiler.  The  second 
group,  static  error  analysis  ...  techniques,  are  designed  to  detect  specific  classes  of 
faults  or  anomalous  constructs  ....  'V^ile  some  types  of  static  error  analysis  can  be 
automated,  others  are  restricted  to  manual  application.  In  contrast,  the  third  group, 
symbolic  evaluation  techniques,  are  entirely  automated.  The  final  group  consists  of 
manual  review  techniques,  namely  code  inspections  and  structured  walkthroughs" 
[YOUN89]. 

These  four  techniques  can  be  generally  grouped  into  manual  analysis  and  automated  (tool 
assisted)  analysis  techniques.  Note  that  the  automated  tools  can  be  vendor  supplied  (commercial-off- 
the-shelf)  or  can  be  developed  for  a  specific  project.  This  report  reviews  several  types  of  source  code 
static  analysis  technologies. 


3.1  Manual  Static  Analysis 

Manual  static  analysis  consists  of  inspections,  reviews  (formal  and  informal),  walkthroughs, 
and  desk  checking.  Watts  Humphrey,  the  originator  of  the  Software  Engineering  Institute's  Software 
Process  Maturity  Model,  expresses  the  importance  of  inspections: 


"Software  inspections  provide  a  powerful  way  to  improve  the  quality  and  productivity 
of  the  software  process ....  Since  we  tend  not  to  see  evidence  that  conflicts  with  our 
strongly  held  beliefs,  our  ability  to  find  errors  in  our  own  work  is  impaired.  Because 
of  this  tendency,  many  programming  organizations  have  established  independent  test 
groups  that  specialize  in  finding  problems.  Similar  principles  have  led  to  software 
inspections.  With  large-scale  complex  programs,  a  brief  inspection  by  competent  co¬ 
workers  invariably  turns  up  mistakes  the  programmers  could  not  have  found  by 
themselves"  [HUMP90]. 

Tom  Gilb  points  out  that  there  are  significant  differences  between  inspections,  walkthroughs, 
and  reviews. 


"Reviews  and  walkthroughs  are  typically  peer  group  discussion  activities  -  without 
much  focus  on  defect  identification  and  correction ....  Walkthroughs  are  generally 
a  training  process,  and  focus  on  learning  about  a  single  document.  Reviews  focus 
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more  on  consensus  and  buy-in  to  a  particular  document ....  Inspection  is  a  quality 
improvement  process  for  written  material.  It  consists  of  two  dominant  components; 
product  (document  itself)  improvement  and  process  improvement  (of  both  document 
production  and  Inspection)"  [GILB93]. 

Desk  checking  is  basically  an  informal,  self-checking  review  technique  that  is  smart  practice 
and  should  be  employed  after  producing  any  kind  of  software  or  documentation.  However,  as  quoted 
from  [HUMP90]  above,  desk-checking  has  a  limited  potential  for  finding  defects  because  of  our 
inability  to  find  all  errors  in  our  own  work.  Note  that  tools  are  being  developed  to  specifically 
support  software  inspections.  These  types  of  tools  are  not  yet  reviewed  in  this  report. 

3.2  Automated  Static  Analysis 

Source  code  static  analysis  tools  are  useful  for  browsing,  measuring,  displaying,  decomposing, 
reengineering,  and  maintaining  source  code  [ARNO90].  Some  examples  of  types  of  tools  that 
support  these  functions  are  listed  beside  Youngblut's  static  analysis  techniques  as  follows: 

•  General  Information  -  cross  referencing  tools,  size  measurers,  complexity  measurers. 

•  Static  Error  Analysis  -  syntax  and  semantic  analyzers,  structure  checkers. 

•  Symbolic  Evaluation  -  symbolic  evaluators,  proofs  of  correctness. 

The  types  of  source  code  static  analysis  tools  that  this  document  reviews  include  those  tools 
that  support  the  General  Information  and  Static  Error  Analysis  techniques.  Section  2.3. 3. a 
introduced  source  code  static  analyzer  tools.  This  section  provides  additional  guidance  and 
motivation  for  using  source  code  static  analyzers.  Note  that  there  are  very  few  products  that  were 
designed  to  perform  only  one  major  static  analysis  application;  most  tools  provide  multiple 
capabilities. 


3.2.1  Auditors 

Source  code  quality  is  a  primary  concern  within  the  software  community.  Due  to  this 
concern,  many  agencies  are  establishing  Total  Quality  Management  (TQM)  policies.  The  idea  of 
quality  software  (or  quality  systems)  is  not  new,  but  with  the  complexity  of  today's  systems,  it  is 
receiving  added  emphasis. 
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Bill  Hetzel  states,  "The  cost  of  quality  falls  into  three  broad  categories:  prevention,  appraisal, 
and  failure"  [HETZ88].  Lack  of  attention  in  the  prevention  and  appraisal  categories  implies,  there's 
never  enough  time  to  do  it  right,  but  there's  always  enough  time  to  do  it  over.  Tight  schedules,  little 
money,  and  the  lack  of  the  appropriate  processes  and  effective  tools  encourage  the  overly  optimistic 
view  that  there  won't  be  defects  and  failures. 

Some  automated  support  is  available  for  analyzing  code  to  ensure  that  there  have  been  no 
violations  of  coding  standards  such  as  variable  naming  conventions,  structuring  conventions,  code 
nesting  depth  checks,  etc.  Auditors  can  play  a  significant  role  in  reducing  the  code  inspection  effort. 
Inspection  checklists  don't  need  to  include  checks  for  characteristics  that  auditors  evaluate.  Source 
code  auditors  determine  adherence  to  coding  standards  by  automatically  comparing  the  developed 
code  to  established  coding  standards. 

3.2.2  Complexity  Measurers 

Years  ago,  the  only  way  of  measuring  the  size  or  complexity  of  code  was  to  count  the  number 
of  lines  of  code.  Today,  there  are  still  many  organizations  that  use  this  as  their  only  method  of 
measurement.  There  is  still  a  lot  of  discussion  about  what  constitutes  a  line  of  code;  are  comments 
to  be  counted?  What  about  blank  lines?  Knowing  the  size  of  a  program  is  still  a  vital  piece  of 
information,  and  it  really  doesn't  matter  what  the  project  (or  company)  determines  constitutes  a  line 
of  code  as  long  as  they  are  consistent.  But  don't  fall  into  the  trap  of  trying  to  compare  your 
productivity,  i.e.,  lines  of  code/month,  against  someone  else's  published  metrics;  you  will  probably 
be  comparing  apples  and  oranges. 

Counting  the  lines  of  code  shows  how  big  the  module  is,  but  gives  little  indication  of  its 
complexity.  Consider  the  following  two  lines  of  code  as  an  example: 

•  c  =  25; 

•  if  (((a  <=  0)  and  ((b  >  c)  or  (b  <  d  and  c  >  b)))  then 

Both  of  the  above  examples  are  a  single  line  of  code,  but  the  complexity  of  the  second  is 
obviously  greater  than  the  first.  Complexity  of  the  software  can  be  considered  how  "busy"  the  code 
is.  The  "busier"  the  code,  the  more  difficult  it  is  to  develop  and  ultimately  to  maintain.  Some  of  the 
benefits  of  measuring  the  complexity  of  the  software  are 
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•  Determining  how  difficult  the  code  will  be  to  test  and  maintain.  This  information  can 
help  when  allocating  testing  resources. 

•  Identifying  which  sections  of  code  should  be  rewritten. 

•  Obtaining  an  estimate  of  the  number  of  defects  to  expect.  Research  has  shown  a 
correlation  between  complex  code  and  defects. 

Complexity  measurers  can  analyze  code  and  identify  potential  problem  areas  that  may  be  too 
difficult  to  effectively  maintain.  Complexity  measurers  can  be  used  as  early  as  the  design  phase  by 
analyzing  program  design  language  (PDL)  and  during  development  to  help  minimize  complexity 
problems  as  early  as  possible. 

The  industry  is  be^nning  to  seriously  use  complexity  metrics  as  additional  input  to  determine 
the  test  schedules.  The  two  most  widely  recognized  complexity  metrics  are  McCabe's  cyclomatic 
complexity  and  Halstead's  software  science  metrics.  McCabe's  complexity  metric  can  be  defined  in 
several  ways,  the  simplest  of  which  is  to  count  the  number  of  decisions  in  a  routine,  and  add  one. 
Knowing  the  cyclomatic  complexity  of  the  code  can  help  the  developer  determine  the  minimum 
number  of  independent  unit  tests  that  are  required  to  execute  every  line  of  code,  and  every  decision 
path,  within  a  module.  Halstead's  complexity  metrics  consist  of  a  series  of  metrics  based  on  the  total 
and  unique  numbers  of  operators  and  operands.  The  metrics  include  theoretical  estimates  of  length, 
program  volume,  program  level,  effort,  etc. 

3.2.3  Cross  Referencing  Tools 

Interfaces  between  software  modules  is  another  problem  that  programmers  have  always  had 
to  deal  with.  Cross  referencing  tools  provide  the  information  so  that  when  a  variable  is  modified  in 
one  section  of  code,  the  programmer  knows  all  other  places  that  the  code  will  have  to  be  modified. 
This  is  also  extremely  helpful  for  the  maintainers  of  the  code.  Some  of  the  capabilities  provided  by 
good  cross  referencing  tools  are 

•  Allows  the  programmer  to  perform  cross-referencing  on  the  entire  system. 

•  Allows  cross-referencing  while  still  within  an  editing  session. 

•  Provides  on-line,  interactive  cross-referencing,  and  viewing  capability. 

Note  that  most  good  compiler  systems  provide  some  cross-referencing  capability. 
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3.2.4  Size  Measurers 

Because  of  the  interest  in  tools  that  measure  SLOC  and  function  points  as  a  result  of  the  new 
Air  Force  Software  Metrics  Policy  93M-017,  size  measures  was  added  to  the  static  analyzer  tool 
classification.  The  metrics  policy  requires  that  all  Air  Force  organizations  and  contractors  that  do 
business  with  the  Air  Force  identify  and  collect  a  minimal  set  of  metrics  that  have  the  attributes  of 
size,  effort,  schedule,  quality,  and  rework  for  each  of  their  projects  that  consist  of  20,000  SLOC  or 
more. 

There  are  several  types  of  size  measurement  tools  including  SLOC  estimators,  functions  point 
estimators,  and  actual  SLOC  measurers.  This  report  focuses  on  measurement  of  existing  code. 
Actual  SLOC  measurers  generally  count  comments,  total  lines,  executable  lines,  and  declarations. 
Some  of  these  tools  provide  historical  data  bases  to  track  code  growth  and  changes.  An  attribute  of 
some  SLOC  measurers  is  the  capability  to  automatically  collect  SLOC  counts  at  predetermined  times. 
Most  code  complexity  measurement  tools  provide  some  SLOC  counts. 

3.2.5  Structure  Checker 

Understanding  the  structure  of  the  software  is  another  problem  that  maintenance  programmers 
must  deal  with.  As  the  code  grows  and  becomes  more  complex  during  development,  a  programmer 
often  has  a  difiScult  time  remembering  which  modules  are  called  by  other  modules.  These  tools  are 
particularly  useful  when  maintaining  or  reengineering  code.  Structure  checkers  provide  increased 
visibility  of  systems  under  development  and  maintenance.  Structure  checkers  report  through  graphic 
or  textual  means  the  module  call  structure.  Structure  checkers  also  analyze  code  structure  for  "dead 
code"  (not  capable  of  being  executed),  recursion  (in  languages  like  FORTRAN  where  it  is 
undesirable),  and  nonstructured  constructs,  e.g.,  GO  TOs.  Note  that  some  code  may  be  logically 
"dead,"  and  can  only  be  identified  by  dynamic  execution  of  the  code. 

3.2.6  Syntax  and  Semantics  Analyzers 

Syntax  and  semantics  analyzers  enhance  the  analysis  capability  of  average  compilers.  Often, 
when  someone  thinks  of  source  code  static  analysis  tools,  what  comes  to  mind  are  the  functions  under 
this  category  of  tools.  Syntax  is  defined  as  the  rules  governing  the  structure  of  a  language,  and 
semantics  is  the  relationship  of  characters  and  their  meanings.  Compilers  check  for  syntax  of  the 
language  to  make  sure  that  the  rules  of  the  language  are  being  followed.  Because  of  the  nature  of 
the  language,  Ada  compilers  perform  more  semantics  checking  than  other  language  compilers. 
Syntax  and  semantic  analyzers  typically  perform  the  following  functions: 
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•  Check  for  uninitialized  variables. 

•  Check  calling  argument  compatibility  between  the  calling  routine  and  the  routine 
being  called. 

•  Check  for  certain  semantic  errors.  For  instance,  a  variable  that  is  defined  and  not 
used. 
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4  Evaluation  and  Selection  of  Test  Technologies 

As  can  be  seen  from  Section  2,  there  are  many  types  of  testing  tools.  There  are  also  many 
test  tool  functions  and  categories  not  listed  in  this  publication.  This  report  provides  evaluation  and 
selection  information  about  the  major  types  of  test  tools.  Some  new  tools  and  testing  technologies 
are  appearing  in  the  marketplace  and  will  be  included  in  future  releases  of  this  report  as  information 
becomes  available  and  as  the  technology  matures  or  becomes  more  prevalent. 

This  section  describes  the  appendices  that  catalog  tool  information,  conferences  and  seminars, 
references,  and  a  glossary  of  testing  terms.  This  section  also  addresses  the  process  for  evaluating  and 
selecting  software  testing  technologies  from  a  wide  assortment  of  test  tool  types  and  their  specific 
offerings.  Many  methods  have  been  devised  to  assist  with  product  evaluation  and  selection.  Most 
of  these  methods  are  difficult  to  follow  because  some  of  the  information  that  they  recommend  is 
difficult  to  obtain.  Even  the  task  of  surveying  tools  can  be  difficult  and  time  consuming  when  starting 
from  scratch.  This  technology  report  provides  a  good  place  to  start  the  evaluation  and  selection 
process  by  providing 

•  Introductory  information  about  testing  technologies  and  practices. 

•  Classification  of  test  tools. 

•  Guidelines  for  evaluating  and  selecting  software  test  tools. 

•  A  catalog  of  candidate  tools. 

•  Guidance  for  improving  the  testing  process  using  the  Software  Engineering  Institute's 
Capability  Maturity  Model. 

•  Test  technology  references,  glossary,  and  conferences. 

Preliminary  or  brief  evaluations  (product  critiques)  from  experienced  practitioners  are  also 
available  upon  request  from  the  STSC.  Finally,  detailed  evaluations  (quantitative  evaluations)  are 
available  upon  request  as  are  the  characteristics  for  performing  the  detailed  evaluations. 


4.1  Test  Information  Catalog 

Appendix  A  contains  a  test  tool  catalog  (list).  Appendix  B  contains  information  about  STSC's 
product  critique  system.  Appendices  C  through  E  contain  additional  information  about  references. 
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testing  conferences  and  seminars,  and  a  glossary  of  testing  terminology,  respectively.  The  following 
sections  further  describe  the  contents  of  the  appendices: 


4.1.1  Test  Tool  Lists 

Test  tools  can  be  identified  by  reading  trade  publications  (including  periodicals,  books,  and 
other  tool  catalogs),  attending  conferences,  and  networking  with  peers.  Appendix  A  catalogs  test 
tools  by  tool  name,  vendor  name,  and  tool  type.  Note  that  some  types  of  test  tools  are  not  listed  in 
this  report  because  of  space  limitations.  A  current  list  of  those  tools  (or  any  t5^e  of  tools  on  which 
the  STSC  has  information)  can  be  obtained  upon  request  from  the  STSC. 

•  Product  Sheets  by  Tool  Name.  The  product  sheets  are  completed  by  the  vendors  and 
include  the  tool  name,  the  version  number,  vendor  name,  phone,  fax.  E-mail  address, 
vendor  address,  point  of  contact,  tool  types  (classifications),  languages  supported,  and 
platforms/operating  systems  supported.  These  sheets  also  contain  product 
descriptions,  pricing,  site  license  availability,  number  sold,  release  date,  product  sheet 
dates,  and  an  indication  of  training,  user  group,  newsletter  availability.  See  Appendix 
A.  1 .  Abbreviated  product  sheets  can  also  be  provided  when  faxing  to  interested 
organizations  to  minimize  the  size  of  the  tool  information. 

•  Test  Tool  Lists  by  Vendor  Name.  Test  tools  are  reported  for  each  vendor  in 
Appendix  A.2. 

•  Test  Tool  Lists  by  Test  Tool  Types.  Test  tools  are  reported  for  each  test  tool  type 
classification  in  Appendix  A.  3. 

Some  other  tool  catalogs  that  are  helpful  include 

•  Software  Quality  Engineering,  Testing  Tools  Reference  Guide,  800-423 -TEST 
[DURA93]. 

•  Software  Maintenance  News,  Software  Maintenance  Technology  Reference  Guide, 
415-969-5522  [ZVEG94]. 

•  CASE  Consulting  Group,  CASE  Outlook,  503-245-6880  [FORT91]. 

•  Grove  Consultants,  CAST  Report,  44-625-616279  (United  Kingdom)  [GRAH93]. 

•  Sentry  Publishing  Company,  Applications  Development  Tools  Product  Guide, 
508-366-2031  [SENT94]. 


4.1.2  Product  Critiques 

Product  critiques  highlight  unique  or  noteworthy  tool  capabilities  and  significant  or  annoying 
problems.  The  STSC  is  soliciting  product  critiques  from  experienced  practitioners  and  providing  this 
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information  upon  request.  Customers  can  read  opinions  from  experienced  colleagues  in  a  similar 
manner  to  the  way  friends  ask  each  other  for  their  opinions  when  buying  a  new  car  or  some  other 
expensive  item  or  service.  Contact  the  STSC  for  product  critiques  of  any  test  tools  that  may  be 
available.  Included  in  Appendix  B  is  a  CrossTalk  article  soliciting  product  critiques  [PETE92].  The 
article  explains  STSC's  product  critique  system  and  gives  instructions  for  completing  the  form.  We 
invite  all  experienced  practitioners  to  participate  in  the  product  critique  system  by  completing  a 
product  critique  form  (included  in  Appendix  B)  for  each  software  development  or  maintenance  tool 
used. 

4.1.3  References 

Appendix  C  contains  cited  references  to  publications  that  discuss  test-related  technologies. 

4.1.4  Testing  Conferences  and  Seminars 

Appendix  D  contains  a  list  of  conferences  and  seminars  that  are  of  particular  interest  to 
software  testers  and  quality  assurance  personnel. 


4.1.5  Glossary 

Appendix  E  contains  a  glossary  of  software  testing  technology  terms. 

4.2  Evaluating  Test  Technologies 

The  STSC  has  adopted  a  simplified  approach  to  tool  evaluation  that  does  not  require  the  use 
of  detailed  test  evaluation  procedures  to  verify  fimctionality  and  determine  quality  characteristics 
about  software  products.  STSC's  recommended  test  tool  evaluation  guidelines  start  by  obtaining  this 
technology  report,  defining  and  funding  the  evaluation  project,  and  proceeding  as  follows; 

(1)  Identify  candidate  technologies  for  information  that  is  desired. 

(2)  Training  is  encouraged  early  for  state-of-the-practice  testing  techniques  to  obtain 
more  information  about  available  testing  technologies  that  are  supported  by 
automated  tools.  Attendance  at  testing  conferences  and  seminars  is  also 
recommended. 

(3)  Perform  needs  analyses  to  document  what  test  technologies  are  needed.  A  defined 
testing  process  is  highly  recommended;  however,  complete  and  accurate  software 
requirements  are  essential  in  determining  specific  testing  needs  for  a  given  system. 
Candidate  tools  need  to  be  identified  and  cost  estimates  obtained.  The  basic  costs 
must  be  estimated  for  tools,  additional  hardware  and  software,  training,  and 
integration  for  the  top  few  candidate  tools.  Estimates  on  the  impact  to  the  current 
processes  are  recommended  for  those  tools  that  require  significant  "ramp-up"  before 
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they  can  be  used  effectively,  e.g.,  coverage  analyzers  may  not  require  much  time  to 
learn  and  use  properly  whereas  requirements-based  test  case  generators  will  require 
training  and  an  associated  "learning  curve." 

(4)  Review  or  request  product  critiques  or  any  available  tool  evaluation  information  on 
hand  at  the  STSC  for  an  initial  look  at  tool  performance  and  market  acceptance. 

(5)  The  top  few  technology  candidates  need  to  be  evaluated  in  more  detail  if  the  tools  are 
a  major  investment  in  terms  of  tool  costs,  support,  training,  and  adoption  into  the 
organization  [FIRT87]  contains  generic  software  tool  questions  important  for  many 
software  development  tools.  [IEEE92]  criteria  lists  also  provide  an  excellent  source 
of  important  evduation  characteristics.  Scores  can  be  computed  in  a  similar  manner 
to  Mosley's  Efficient  Tool  Assessment  Methodology  [MOSL92]  (see  Section  4.3.8) 
or  you  can  use  the  weighted  system  provided  in  [IEEE92].  Evaluations  are 
performed  with  the  user's  needs  in  mind  to  minimize  the  evaluation  effort.  This 
process  ensures  that  efforts  and  expenditures  are  minimized  for  each  evaluation  while 
providing  the  most  current  available  data. 

(6)  DoD  organizations  that  require  detailed  (quantitative)  evaluations  should  contact  the 
STSC.  Funding  for  quantitative  evaluations  generally  would  come  from  the  DoD  user 
(requesting)  organization.  Costs  could  be  amortized  depending  on  user  interest  in  the 
evaluation  information.  Expert  tool  users  would  then  be  identified  from  the  product 
critique  authors  and  would  be  invited  to  perform  quantitative  evaluations  based  on 
known  user  needs.  Availability  of  evaluators  will  need  to  be  negotiated  between  the 
DoD  user  organization,  the  STSC,  and  the  evaluating  organization. 

Evaluators  will  be  provided  lists  of  characteristics  that  specify  the  criteria  for  tool 
evaluation  and  scoring.  More  than  one  evaluation  of  a  given  tool  will  be  sought,  and 
discrepancies  will  be  resolved.  Lists  of  evaluation  characteristics  can  be  provided 
upon  request,  which  cover  issues  such  as  ease  of  use,  power,  robustness,  functionality 
of  the  various  types  of  tools,  ease  of  insertion,  and  the  quality  of  vendor  support. 

(7)  Depending  on  the  amount  of  the  total  investment,  a  special  test  case  may  need  to  be 
developed.  Each  tool  or  technology  would  then  be  evaluated  by  potential  users 
(technology  champions)  as  to  how  well  it  supports  their  testing  effort.  Special 
emphasis  should  be  placed  on  evaluating  user  interfaces,  which  may  be  the  only 
significant  difference  between  various  types  of  testing  tools.  For  example,  candidate 
coverage  analyzers  may  perform  the  same  basic  functions,  but  the  number  of  required 
commands  and  the  conciseness  of  the  outputs  may  make  one  tool  far  superior  to 
another. 


4.3  Selecting  Test  Technologies 

Before  selecting  and  installing  the  tool,  the  organization  needs  to  consider  the  impact 
that  the  tool  will  make  on  its  process.  Meaningful  metrics  need  to  be  accumulated  for  measuring  the 
performance  and  quality  improvements.  Even  if  a  tool  is  not  selected  and  implemented,  organizations 
should  begin  collecting  metrics  that  are  meaningful  for  them,  and  organizational  progress  should  be 
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monitored  as  required  by  the  new  Air  Force  Software  Metrics  Policy  [DRIJY94].  The  following 
issues  need  to  be  considered  when  selecting  automated  software  test  technologies; 

•  Test  processes 

•  Test  lifecycle 

•  Budgets 

•  Personnel 

•  Schedules 

•  Project  issues 

•  Training  requirements  associated  with  test  tools 

•  Tool  scoring  techniques 

Manual  testing  is  defined  as  the  execution  of  the  developed  software  program  or  system 
through  the  normal  user  interface  and  manually  observing  results.  Computer-assisted  or  automated 
testing  is  defined  as  the  use  of  software  test  tools  to  automatically  perform  one  or  more  testing 
functions. 

4.3.1  Test  Processes 

According  to  the  Software  Engineering  Institute  (SEI),  all  software  development 
organizations  should  establish  a  Software  Engineering  Process  Group  (SEPG).  This  group  may 
consist  of  one  technology  champion  or  several  representatives  of  an  organization's  development  team 
who  would  be  charged  with  the  responsibility  of  finding  ways  to  improve  software  development 
processes.  New  technologies,  including  methodologies,  techniques,  and  toots,  should  be  researched 
and  appropriately  inserted  in  a  planned,  orderly  fashion. 

Obviously,  if  an  organization's  software  development  process  has  instituted  the  use  of  selected 
automated  test  tools,  then  those  tools  should  be  used.  When  any  development  phase  is  upgraded  with 
new  technologies,  the  testing  process  must  be  reconsidered  since  all  phases  involve  some  form  of 
testing.  When  new  CASE  technologies  are  inserted  to  support  requirements  analysis  and  design, 
testing  strategies  may  need  to  be  reworked  throughout  the  development  lifecycle. 
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An  important  (but  easily  overlooked)  advantage  to  automated  testing  is  the  increase  in 
technological  sophistication  within  the  organization  that  comes  with  the  use  of  automated  tools.  The 
benefits  of  this  increase  in  skills  may  be  difficult  to  quantify,  but  technological  sophistication  builds 
on  itself  Cultivating  the  required  technical  skills  is  a  necessary  element  of  process  improvement. 
Fundamentally,  a  testing  methodology  needs  to  be  recognized  and  uniformly  adopted  by  the 
organization  before  supporting  tools  can  be  integrated  into  the  testing  process;  otherwise,  the  tools 
may  be  viewed  as  an  interference  to  the  status  quo. 

An  organization  waiting  for  the  perfect  set  of  tools  or  for  prices  to  come  down  may  become 
less  competitive  in  its  ability  to  efficiently  and  effectively  perform  its  testing  roles.  Many 
organizations  will  not  use  some  of  the  new  automated  tools  on  the  market  today.  Organizations  that 
have  acquired  and  used  these  tools  have  a  major  productivity  advantage  over  those  that  do  not. 

4.3.2  Test  Lifecycle 

Since  so  many  software  test  activities  are  on  the  critical  path  of  the  development  schedule, 
test  development  should  be  scheduled  and  accomplished  as  early  as  possible  to  be  prepared  for  unit 
testing,  integration  testing,  system  testing,  and  acceptance  testing.  This  implies  that  the  test  effort 
and  software  development  effort  should  be  started  concurrently,  and  that  a  software  test  development 
lifecycle  should  be  identified  and  coordinated  with  the  software  development  lifecycle  (for  example, 
see  Figure  2-1).  Static  analysis  activities  apply  to  all  phases  of  the  software  development  lifecycle. 
One  published  estimate  reports  that  50  percent  or  more  of  the  software  errors  are  due  to  incorrect 
or  modified  requirements  specifications.  It  is  a  well-known  fact  that  software  reviews  can 
significantly  reduce  the  number  of  errors  in  the  later  phases  of  development.  Testers  should  be 
involved  in  early  analysis  and  design  reviews. 

4.3.3  Budgets 

Short-term  and  long-term  interests  will  most  likely  clash  with  regard  to  budgets.  In  many 
organizations,  management  requires  quantitative  estimates  of  the  retum-on-investment  (ROI)  for  new 
software  tool  purchases.  These  are  often  difficult  to  obtain  without  knowing  the  costs  of  software 
to  start  with.  Sometimes  subjective  justifications  are  all  we  have  to  help  persuade  management  to 
purchase  a  new  tool.  Note  that  it  may  be  easier  to  justify  large  initial  costs  if  the  tools  and  training 
can  be  used  on  other  projects.  This  requires  long-term  strategic  planning  that  spans  many  projects 
and  organizations. 
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4.3.4  Personnel 

An  often  overlooked  point  is  that  the  test  development  effort  is  just  that:  a  development 
effort,  requiring  personnel  with  development  skills,  tools,  and  positive  attitudes.  Test  plans,  test 
cases,  test  software,  and  test  documentation  are  developed.  Software  testers  have  to  be  very  creative 
in  devising  ways  to  increase  test  effectiveness  while  reducing  effort.  Boris  Beizer  has  a  good  list  of 
the  essential  characteristics  of  a  good  tester  in  his  book.  Software  Testing  Techniques  [BEIZ90].  A 
few  of  these  characteristics  include  not  easily  intimidated,  has  integrity,  has  tenacity,  is  detail 
conscious  and  goal  oriented,  and  must  have  a  certain  amount  of  "cop"  in  them.  Testers,  like  cops, 
are  rarely  rewarded  openly  and  directly  for  a  good  job.  If  they  do  a  good  job,  it  can  easily  mean 
added  work  or  embarrassment  for  someone  else. 

There  is  no  question  that  the  more  sophisticated  tools  will  require  an  attendant  increase  in 
skills.  Application  domain  knowledge  and  mature  development  experience  provide  significant 
advantages  for  testers  in  building  effective  test  cases.  Developers  and  testers  then  have  better 
communications  and  confidence  in  each  other.  However,  the  tendency  is  often  to  throw  together  an 
available  group  of  varied  skills  and  personalities  and  call  it  the  test  group.  This  lack  of  personnel 
consideration  will  likely  result  in  a  project  with  many  difficult  problems. 

4.3.5  Schedules 

Technology  acquisition  schedules  and  project  planning  schedules  need  to  be  considered  when 
selecting  new  test  technologies. 

4.3.5. 1  Technology  Acquisition 

If  schedules  do  not  allow  enough  time  to  select  the  proper  new  tools  and  obtain  the  required 
training,  current  techniques  must  be  used.  Ad  hoc  development  and  manual  testing  may  be  effective 
on  very  small  projects  with  the  right  personnel.  But  if  investments  in  effective  technologies  are  not 
made  on  large  projects,  the  organization  will  probably  pay  for  it  during  maintenance. 

4.3. 5.2  Project  Planning 

Whether  automated  test  tools  or  manual  testing  techniques  are  used,  test  planning  should 
begin  as  early  in  the  development  lifecycle  as  possible,  i.e.,  during  requirements  definition.  Testing 
must  be  treated  and  scheduled  as  an  integral  part  of  the  development  effort.  Strong  leadership 
commitment  is  required  to  elevate  the  importance  of  test  preparation  if  an  organization  is  still  not 
committing  test  resources  until  after  code  is  produced. 
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4.3.6  Project  Issues 

In  large  or  complex  projects,  computer-aided  testing  may  be  mandatory  to  automatically 
generate  test  cases,  to  keep  track  of  configurations,  to  automatically  retest  changed  components,  etc. 
Requirements  tracing  can  easily  get  out  of  hand  without  modem  tools.  Manual  checking  of  design 
and  code  structure  can  become  burdensome.  How  do  you  stress  test  a  network  that  has  the  capacity 
to  support  several  hundred  nodes?  How  do  you  test  this  network  continuously  over  several  days? 
If  the  inputs  and  responses  happen  too  fast  for  a  human  to  keep  up,  then  computer-assisted  testing 
also  is  mandatory. 

Employee  turnover  can  cripple  a  project  if  a  mechanism  is  not  in  place  to  retain  as  much 
project  knowledge  as  possible.  It  is  very  likely  that  key  personnel  will  leave  during  a  long-term 
project  and  take  their  skills  with  them.  Some  automated  testing  tools  can  be  thought  of  as  skills 
repositories  to  preserve  critical  capabilities  during  the  life  of  the  product.  People,  for  a  variety  of 
reasons,  cannot  be  relied  upon  to  completely  remember  or  document  all  development  or  test  details. 
Capture-replay  tools  can  record  all  test  inputs  (document  automatically  or  permit  editing  of  test 
scripts).  Tests  can  be  mn  automatically  and  the  results  can  be  compared  automatically  to  the 
expected  outputs  with  no  human  intervention. 

It  may  be  difficult  to  justify  the  cost  in  time  and  money  of  automated  test  tools  and  training 
if  the  product  is  a  one-of-a-kind  or  one-time-only  product,  unless  the  project's  complexity,  size,  and 
precision  requirements  or  the  future  plans  for  tools  dictate  automated  testing. 

4.3.7  Training  Requirements  Associated  with  Test  Tools 

Along  with  the  added  sophistication  of  new  technology,  come  the  added  costs  in  time  and 
money  for  the  training  necessary  to  use  this  technology.  There  is  no  easy  way  around  this.  The 
learning  curve  may  require  that  more  immediate  projects  not  be  included  in  the  new  development 
strategy.  The  cost  of  this  technology  can  be  amortized  over  a  large  number  of  projects.  There  is  a 
good  chance  that  test  automation,  incorporated  with  a  proper  level  of  software  inspections  and 
reviews,  will  reduce  test  and  rework  efforts  and  budgets  over  time  while  providing  higher  quality 
systems. 

A  hasty,  ill-considered  tool  selection  will  be  a  costly  burden  to  bear  for  a  long  time.  This  is 
not  a  new  problem.  A  word  processor  or  database  management  system  that  is  not  carefully  selected 
can  be  a  long-time  thorn  in  productivity's  side.  The  game  is  the  same,  but  the  stakes  are  getting 
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higher  in  the  software  development  world.  The  costs  of  a  bad  tool  selection,  in  retraining  time  alone, 
may  doom  an  entire  product,  organization,  or  company.  Sophisticated  technology  is  a  double-edged 
sword  that  can  help  you  or  hurt  you. 

Development  support  activity  training  (test  in  particular)  is  usually  low  on  most  priority  lists. 
Training,  or  more  aptly  retraining,  can  be  the  winning  or  losing  edge  in  the  sophisticated  software 
world  of  which  we  are  a  part.  Training  cannot  be  treated  as  an  afterthought.  There  is  too  much  at 
stake. 

4.3.7. 1  Formal  Training 

Software  testing  is  not  a  subject  that  is  commonly  taught  at  the  college  level  as  a  separate 
subject.  If  taught  at  all,  it  is  usually  included  as  part  of  the  general  software  engineering  courses. 
There  are  a  few  specific  software  testing  classes  to  be  found,  however,  and  probably  more  to  come 
as  quality  becomes  a  hotter  topic. 

4.3. 7.2  Vendor  Training 

Vendors  will  usually  supply  some  kind  of  training.  The  main  drawback  here  is  the  costs  for 
time,  classrooms,  and  manuals  (printed  or  electronic).  Customer-site  training  is  often  available  and 
depending  on  the  number  of  people  involved  can  be  cost-effective.  Otherwise,  individuals  can  go  to 
the  vendor  site;  however,  this  requires  significant  resources.  Vendor  manuals  are  the  most  common 
form  of  vendor  training.  The  learning  process  is  slower,  but  you  may  learn  just  what  you  need  to  get 
the  job  done.  Some  vendors  provide  on-line  tutorials  or  computer-based  instruction  to  help  reduce 
the  costs  of  training  while  broadening  the  base  of  potential  trainees. 

4.3. 7.3  Seminars,  Conferences,  Meetings,  and  Workshops 

Seminars,  conferences,  meetings,  and  workshops  are  taught  at  more  of  a  generic  level,  unless 
the  tool  in  question  is  very  common  and  possibly  dominates  its  tool  domain.  These  courses  can 
sometimes  be  more  informative  than  those  supplied  by  the  vendor.  Costs  usually  reflect  this  reality. 
Some  seminars  are  built  around  a  particular  method  or  tool  that  is  currently  getting  generous  press 
coverage.  Seminars  are  good  for  learning  about  new  technologies  and  are  an  effective  way  of 
obtaining  contacts  within  the  industry.  Government  and  industry  groups  offer  a  wide  variety  of 
testing  conferences  that  deal  with  all  aspects  of  development  and  testing. 
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Appendix  D  and  the  STSC's  periodical  publication,  CrossTalk,  contain  lists  of  upcoming 
conferences  and  seminars  that  may  be  of  interest  to  readers. 


4.3.7.4  Individual  Research 

By  far,  the  most  common  forms  of  education  in  the  computer  world,  and  testing  in  particular, 
are  independent  reading  and  consulting  with  peers.  Industry  trade  journals  abound  and  cover  nearly 
every  subject  imaginable.  Many  publications  are  free  if  the  reader  is  employed  in  that  subject  area, 
but  others  can  be  costly  and  should  be  bought  by  an  organization's  library  for  everyone's  benefit. 
Relations  with  peers  are  valuable  and  should  be  nurtured.  Often,  there  are  personnel  in  the  immediate 
organization  that  have  first-hand  experience  who  can  and  should  be  tapped  for  information. 


4.3.8  Tool  Scoring  Techniques 

All  of  the  tool  selection  sections  under  Section  4.3  impact  a  tool  selection  decision  to  a 
greater  or  lesser  degree.  After  consideration  of  these  issues,  it  makes  sense  to  compare  similar  tools 
to  identify  strengths  and  weaknesses  based  on  the  detailed  evaluation  scores  obtained  by  following 
the  guidelines  in  Section  4.2.  The  STSC  has  defined  spreadsheets  that  automatically  sum  up  weighted 
evaluation  scores.  A  sample  scoring  spreadsheet  or  matrix  (with  fictitious  data)  is  presented  in 
Figure  4-1.  Features  of  the  Tool  Scoring  Matrix  include 


•  All  of  the  first-level  descendent  characteristics  are  listed  prior  to  listing  lower-level 
descendants.  This  makes  it  easy  to  view  the  important  issues  rather  than  having  to 
potentially  scan  several  pages  to  find  scores  for  those  first-level  descendants.  Also, 
briefing  charts  can  be  easily  prepared  to  report  the  most  significant  information. 

•  Only  ones  and  zeros  were  used  at  the  lowest-level  descendent  characteristics  to 
indicate  whether  a  tool  has  a  capability  or  not.  (The  score  for  a  characteristic  was  left 
blank  if  it  was  not  evaluated.)  Any  scoring  range  could  have  been  used  (depends  on 
user  desires)  but  you  may  want  to  keep  the  scoring  range  simple. 

•  Weights  can  be  defined  according  to  user  desires;  however,  assigning  weights  that  are 
not  equal  at  low-level  groups  is  usually  subjective  and  may  be  difficult  to  obtain 
consensus.  The  sum  of  the  individual  characteristic  weights  appears  on  the  top  line 
of  the  group  under  the  weight  column. 

•  Group  scores  consist  of  the  summation  of  the  weighted,  normalized  scores  (between 
0  and  1)  of  the  descendent  characteristics. 

Call  the  STSC  for  information  about  computing  scores  and  selecting  tools  based  on  evaluation 
information. 
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A  Comparative  Evaluation  of  Tool_A  and  Tool  B  3i  May  94 


ID 

CHARACTERISTICS 

TOOL_A 

TOOL_B 

WEIGHTS 

Comulexitv  Measurer 

0.80 

0.90 

1 

1 

Ease  of  Use 

0 

2 

Power 

0 

3 

Robustness 

0 

4 

Functionality 

0.80 

0.90 

1 

5 

Ease  of  Insertion 

0 

6 

Quality  of  Vendor  Support 

0 

7 

Other 

0 

4 

Functionality 

0.80 

0.90 

2 

4.1 

Methodology  Support 

4.2 

Correctness 

4.3 

Types  of  Metrics 

0.60 

0.80 

1 

4.4 

Types  of  Reports 

4.5 

Database  Support 

1 

1 

1 

•  •• 

4.3 

Tvnes  of  Comnlexitv  Metrics 

0.60 

WKm 

5 

4.3.1 

McCabe  Metrics 

1 

1 

1 

4.3.2 

Halstead's  Metrics 

0 

1 

1 

4.3.3 

Nesting  Depth 

1 

1 

1 

4.3.4 

Variable  Span  of  Reference 

0 

1 

1 

4.3.5 

Design  Structure 

1 

0 

1 

Figure  4-1.  Sample  Tool  Scoring  Matrix. 
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5  Improving  the  Test  Process  Using  the  CMM 

Last  year's  test  technology  report,  [TPEE93],  mentioned  in  its  Section  5.2  that  the  next 
updated  test  technology  report  would  provide  additional  testing  guidance  at  the  Software  Engineering 
Institute  (SEI)  Capability  Maturity  Model  (CMM)  Level  2.  This  section  contains  guidance  for 
improving  test  management  practices  at  Level  2  and  some  test  engineering  practices  at  Level  3 
through  5.  This  section  identifies  testing  (or  test  supporting)  activities  that  exist  in  the  CMM  and 
recommends  some  supplemental  testing  activities  based  on  current  effective  industry  practices. 

5.1  CMM  Background 

The  CMM  provides  a  formidable  tool  that  software  development^  organizations  can  use  to 
improve  their  software  processes.  It  is  a  software  process  maturity  fi'amework  "that  describes  key 
elements  of  an  effective  software  process"  [PAUL93].  The  CMM  defines  five  maturity  levels: 

(1)  Initial  -  Ad  hoc,  occasionally  chaotic  process. 

(2)  Repeatable  -  Disciplined  process. 

(3)  Defined  -  Standard  consistent  process. 

(4)  Managed  -  Predictable  process. 

(5)  Optimizing  -  Continuously  improving  process. 

Each  maturity  level  is  composed  of  several  Key  Process  Areas  (ICPAs),  which  are  related 
activities  "that,  when  performed  collectively,  achieve  a  set  of  goals  at  that  maturity  level"  [PAUL93]. 
Also,  the  maturity  levels  suggest  an  orderly  progression  to  develop  effective  management  practices 
at  Level  2,  then  improve  technical  and  organizational  practices  at  Level  3.  With  the  foundation  of 
the  previous  levels.  Level  4  initiates  statistical  processes  through  establishing  improved  measurements 
and  process  improvement  goals  and  finally.  Level  5  focuses  on  continuous  process  improvement 
through  defect  prevention  and  improved  practices  and  technologies. 

5.1.1  CMM  Testing  Issues 

Software  test  management  practices  need  to  be  reviewed  early  in  a  process  improvement 
initiative.  The  CMM  tells  us  that  until  sound  management  practices  are  in  place,  such  as  those 


^  Software  development  also  refers  to  software  maintenance  in  this  section. 
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identified  at  the  Repeatable  Maturity  Level,  "the  benefits  of  good  software  engineering  practices  are 
undermined  by  ineffective  planning  and  reaction-driven  commitment  systems"  [PAUL93]. 

However,  there  are  some  software  testing  issues  that  need  additional  attention  in  the  CMM, 
particularly  in  the  area  of  test  management.  Modern  testing  principles  recognize  that  software  test 
development  parallels  software  development.  Software  test  development  has  a  lifecycle  that  starts 
with  planning,  proceeds  through  analysis  of  test  objectives  (test  requirements)  and  design  of  test 
cases,  then  follows  with  implementation  (building  test  scripts,  test  data,  etc.).  Evaluation  of  the  test 
work  products  (testware:  test  plans,  test  procedures,  test  cases,  etc.)  occurs  at  each  test  lifecycle 
phase.  Some  test  management  and  planning  issues  are  either  not  addressed  in  the  CMM  or  are 
embedded  in  the  CMM  in  several  general  development  guidelines  that  do  not  specifically  address 
testing.  This  can  make  them  difficult  to  apply  to  improve  software  testing. 

5.1.2  Example  Concerns 

The  Repeatable  Maturity  Level  does  not  require  the  process  to  have  defined  test  objectives 
nor  a  specific  test  plan  to  address  those  objectives.  Proper  management  of  test  development  requires 
defined  test  objectives.  Software  requirements  cannot  be  exhaustively  tested  under  all  data  input 
conditions  and  process  states;  therefore,  test  objectives  are  required  to  define  how  conformance  to 
requirements  will  be  evaluated.  Test  objectives  include  functions  and  constraints,  software  states, 
input  and  output  data  conditions,  and  usage  scenarios  to  test  for  at  the  various  testing  levels  used  on 
the  project,  e.g.,  unit,  integration,  system,  acceptance  [GELP94].  As  an  industry,  we  have  learned 
the  importance  of  defining  and  managing  requirements.  We  don't  want  to  be  making  fundamental 
requirements  and  design  decisions  while  we're  coding  the  system;  similarly,  we  don't  want  to  be 
making  decisions  about  test  objectives  and  test  designs  while  we're  building  (coding)  the  test 
procedures  and  test  data. 

Obtaining  early  agreement  from  the  customer  organization  on  the  test  objectives  can  avert 
many  problems,  such  as  unprepared  and  uninformed  testers  with  no  testing  goals,  that  can  otherwise 
occur  near  the  end  of  a  project.  Test  objectives  defined  early  can  affect  and  actually  guide 
requirements  definition  and  analysis,  design,  and  coding.  Test  cases,  including  success  criteria,  are 
designed  fi'om  the  test  objectives.  Test  objectives  are  also  required  to  effectively  track  test  status  and 
repeat  progress  on  future  projects. 
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Determining  test  objectives  is  a  technical  activity,  but  managing  those  objectives  is  a  test 
management  activity.  Unfortunately,  a  common  perception  is  that  most  test  activities  are  engineering 
activities  to  be  addressed  at  Level  3,  and  they  do  not  need  to  be  considered  in  the  early  stages  of 
process  improvement.  A  CMM-Based  Appraisal  for  Internal  Process  Improvement  (CBA  DPI), 
formerly  called  Software  Process  Assessments  (SPAs),  may  ignore  test  management  issues  in  the 
resulting  process  improvement  Action  Plans.  Some  software  management  principles  in  the 
Requirements  Management  KPA  need  to  be  applied  specifically  to  test  development.  Section  5 .3 . 1 .2 
lists  some  recommended  test  management  practices  for  managing  test  objectives. 

5.2  CMM  Benefits 

More  organizations  are  recognizing  the  benefits  of  using  the  CMM  to  improve  their  software 
development  practices.  This  section  discusses  several  important  benefits  of  using  the  CMM  to 
improve  software  processes,  specifically  testing  processes.  Organizations  should  focus  their 
improvement  efforts  on  "the  few  issues  most  critical  to  software  quality  and  process  improvement" 
[PAUL93].  Improved  testing  practices  can  provide  significant  software  quality  and  process 
improvements.  The  following  sections  discuss  several  important  benefits  of  using  the  CMM  to 
improve  software  testing  processes. 

5.2.1  Advocate 

The  CMM  is  an  advocate  for  software  developers  and  testers  who  have  managers  with 
unreasonable  expectations.  The  CMM  is  also  an  advocate  for  managers  who  have  staff  members  who 
practice  ad  hoc  methods  with  no  accountability.  Top-level  DoD  management  is  committed  to 
achieving  higher  software  process  maturity  levels  |MOSE93].  Developers,  testers,  and  managers  will 
have  a  greater  respect  and  appreciation  for  planning  and  monitoring  progress  with  the  adoption  of 
CMM  practices. 

The  CMM  advocates  senior  management  sponsorship,  written  organizational  policies,  proper 
levels  of  training  for  each  activity,  and  resource  availability  to  do  the  job.  If  any  of  these  controls  are 
missing,  the  project  is  at  risk  and  will  be  difficult  to  complete  on  time,  within  budget,  and  with 
acceptable  quality.  As  an  example,  Raymond  Dion  of  Raytheon  expressed  the  importance  of  training 
in  his  organization,  "Project  managers  who  once  insisted  that  all  training  be  fimded  by  overhead  are 
now  accepting  bids  that  include  training  costs,  because  we  can  demonstrate  a  direct  benefit  to  the 
project"  [DION93]. 
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5.2.2  Structure 

The  CMM  has  an  interesting  horizontal  and  vertical  structure  that  encourages  solving  related 
problems  at  each  level  in  priority  order  as  well  as  evolving  software  development  practices  from  level 
to  level.  While  the  CMM  says  that  "key  process  areas  have  been  defined  to  reside  at  a  single  maturity 
level,"  many  software  practices  evolve  as  the  organization  matures  [PAUL93].  For  example,  defect 
handling  practices  mature  from  policies  for  dealing  with  problems  at  Level  2,  to  documented  practices 
for  tracking  and  analyzing  defects  at  Level  3,  to  quantitatively  predicting  defects  at  Level  4,  to  defect 
prevention  at  Level  5. 

We're  encouraged  to  adopt  practices  that  improve  our  maturity  to  the  next  level  since  they 
provide  the  foundation  for  fiirther  improvements.  However,  some  high-level  practices,  like  setting 
up  a  software  engineering  process  group  (SEPG)  which  is  a  Level  3  activity,  exert  a  maturity-pull 
effect  for  Level  1  organizations.  "They  can  facilitate  the  introduction  and  acceptance  of  other 
practices  and  so  should  be  introduced  early"  [CORD93].  Cord's  theme  is  that  defect-causal  analysis 
(a  Level  3-5  activity)  also  exerts  a  maturity-pull  effect  for  Level  1-2  organizations.  Being  careful  not 
to  take  on  more  than  we  can  handle,  adopting  improved  practices  (acting  mature)  is  the  way  we  start 
to  improve  our  processes. 

Note  that  we  have  to  be  careful  to  adopt  enough  of  a  recommended  practice  in  order  not  to 
develop  bad  habits.  For  example,  if  you  don't  consider  your  organization's  optimum  inspection'*  rate 
when  first  introducing  peer  reviews  or  inspections,  you  will  have  limited  success.  For  peer  reviews 
to  be  effective,  you  need  to  monitor  your  review  rates  to  determine  and  abide  by  your  optimum  rates 
[GILB93].  Measuring  optimum  inspection  rates  is  basically  a  Level  4  activity,  but  it  is  important 
when  first  implementing  peer  reviews. 

The  internal  structure  of  each  KPA  helps  you  consider  important  issues  such  as  your 
organization's  ability  to  do  the  job  and  relevant  configuration  management,  quality  assurance, 
evaluation,  and  progress  measurement  activities  to  ensure  a  good  job  is  done.  Each  KPA  contains 
an  opening  discussion  that  describes  its  purpose.  Then,  specific  goals  of  the  ICPA  are  listed.  Finally, 
a  set  of  Common  Features  are  provided  that  address  the  following: 


*  Further  references  to  peer  reviews  include  inspections. 
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•  Commitment  to  perform  with  required  actions  to  ensure  the  process  is  established. 

•  Ability  to  perform  with  preconditions  that  must  exist. 

•  Activities  performed  with  roles  and  procedures  to  implement  the  KPA. 

•  Measurement  and  analysis  with  associated  example  measurements. 

•  Verifying  implementation  with  steps  to  ensure  compliance. 


5.2.3  Other  Test  Support  Activities 

The  CMM  has  recognized  the  importance  of  requirements,  which  may  be  reflected  in  the 
placement  of  the  Requirements  Management  KPA  as  the  first  KPA  in  the  Repeatable  Maturity  Level. 

The  CMM  includes  peer  reviews  in  addition  to  internal  reviews  with  management  and  formal 
technical  reviews  with  the  customer.  Peer  reviews  have  been  accepted  as  an  important  industry 
practice  that  cost-effectively  identifies  many  defects  in  the  phase  they  were  created  and  effectively 
prevents  them  from  being  passed  on  to  subsequent  phases.  If  any  software  or  testware  is  worth 
building,  it  is  worth  planning,  analyzing,  designing,  implementing,  and  evaluating  similar  to 
production  software.  Note  that  managers  are  not  included  in  peer  reviews,  which  helps  improve  the 
effectiveness  of  finding  defects  in  an  ego-less,  team  environment  during  a  specific  development  phase. 

The  CMM's  key  practices  were  written  "in  terms  of  what  is  expected  to  be  the  normal 
practices  of  organizations  that  work  on  large,  government  contracts"  [PAUL93].  Teamwork  is 
emphasized  along  with  practices  that  improve  the  planning,  tracking,  and  oversight  management 
capabilities  of  the  organization. 

The  CMM  has  a  fiiture.  The  Software  Engineering  Institute  has  a  CMM  Configuration 
Control  Board  that  entertains  suggestions  for  improvements.  Future  editions  may  begin  to  address 
technical  and  human  resource  maturity  issues.  The  next  release  of  the  CMM  may  appear  in  1996. 

5.3  Concerns 

An  organization  improving  its  software  development  practices  will  want  to  consider  the 
testing  process  in  light  of  all  of  its  problems  in  a  priority  order.  Unfortunately,  the  CMM  does  not 
focus  much  on  software  test  management  and  planning,  especially  at  the  Repeatable  Maturity  Level. 
Many  fundamental  test  planning  activities  are  not  addressed  until  Level  3.  The  following  sections 
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address  software  test  management  and  test  engineering  concerns.  Each  section  is  fiirther  divided  into 
subject  areas  that  identify,  discuss,  and  recommend  improvements  for  specific  concerns  about  CMM 
test  practices  or  concepts  that  need  additional  consideration. 

5.3.1  Software  Test  Management  Concerns 

Improving  test  planning,  estimating,  and  tracking  practices  Avill  give  management  increased 
visibility  and  control  over  our  test  development,  execution,  and  analysis  activities  [ROYE93].  As 
mentioned  earlier,  the  CMM  does  address  some  test  management  activities.  However,  it  is  difficult 
to  get  a  clear  picture  of  the  CMM's  recommendations  because  test  management  guidelines  are 
embedded  within  several  general  software  development  practices.  Many  of  those  practices  don't 
specifically  address  testing  nor  do  they  show  enough  software  test  related  examples.  Beizer  states, 

"Although  programmers,  testers,  and  programming  managers  know  that  code  must 
be  designed  and  tested,  many  appear  to  be  unaware  that  tests  themselves  must  be 
designed  and  tested  -  designed  by  a  process  no  less  rigorous  and  no  less  controlled 
than  used  for  code"  [BEIZ90]. 

With  this  in  mind,  a  Software  Test  Management  KPA  is  proposed  to  supplement  the  CMM 
at  the  Repeatable  Maturity  Level.  Subject  areas  as  important  as  software  configuration  management 
and  software  quality  assurance  warrant  particular  attention  in  their  KPA.  Since  software  testing 
generally  involves  a  separate  organization  (except  for  unit-level  testing)  and  because  it  has  separate 
goals  and  concerns  from  software  building  activities,  a  separate  KPA  might  better  address  the  needs 
of  managing  test  development  eftforts.  Also,  because  separate  plans  are  often  required  for  the  various 
testing  levels,  managing  the  development  of  those  plans  needs  to  be  clearly  outlined  and  a  Software 
Test  Management  KPA  could  help  provide  the  needed  attention  to  this  important  activity. 

The  business  of  writing  an  additional  KPA  may  seem  unnecessary  or  even  contradictory  with 
the  CMM;  however,  note  for  example,  that  there  is  no  Database  Management  KPA  because  it  is  not 
a  universal  concern  to  the  industry^  If  you  need  a  Database  Management  KPA,  the  SEI  recommends 
that  you  develop  one  to  support  your  process. 

The  following  subject  areas  identify  major  CMM  software  test  management  concerns.  The 
recommendations  in  these  subject  areas  form  a  framework  for  a  Software  Test  Management  KPA  that 


^The  Software  Subcontract  Management  KPA  also  does  not  universally  apply  but  was  included  because  of  its 
importance.  Note  that  there  are  several  practices  expected  of  subcontractors  that  are  not  expected  of  developers. 
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could  be  used  by  all  organizations  to  better  support  their  testing  practices.  This  framework  of 
recommended  practices  parallels  several  CMM  Level  2  practices  but  specifically  addresses  the  test 
group,  the  test  activities,  and  the  test  work  products. 

5. 3. 1.1  Risks 

Testing  is  not  associated  with  risk  assessment  in  the  Repeatable  Maturity  Level.  Risks  and 
available  resources  dictate  the  scope  of  testing.  Identifying  risks  is  the  heart  of  testing  and  determines 
test  objectives.  If  you  believe  something  is  a  risk,  you  test  for  it.  If  you  want  a  group  of  people  to 
agree  about  when  to  stop  testing,  then  consensus  on  risk  assumptions  must  be  obtained  early  in  the 
development  effort.  This  is  fiindamental  to  the  management  of  the  project.  "This  makes  the  testers 
job  doable"  and  increases  test  effectiveness  [GELP94].  Also,  the  risks  and  test  strategy  will  mold  the 
test  objectives  and  is  fundamental  to  planning  and  managing  the  testing  effort.  The  Software  Test 
Management  KPA  should  require  that  test  objectives  and  test  plans  be  based  on  assessed  risks. 

5.3.1 .2  Test  Objectives 

As  mentioned  in  Section  5.1.2,  Example  Concerns,  test  objectives  are  not  adequately 
addressed  at  the  Repeatable  Maturity  Level.  Some  example  recommended  Software  Test 
Management  key  practices  to  manage  test  objectives  (derived  from  the  Requirements  Management 


KPA)  include 

Goal  1 

Test  objectives  are  controlled  to  establish  a  baseline  for  testing  software. 

Commitment  1 

The  project  follows  a  written  organizational  policy  for  managing  test  objectives. 

Ability  1 

For  each  project,  responsibility  is  established  for  analyzing  software 
requirements  and  risks  to  determine  test  objectives. 

Ability  2 

The  test  objectives  are  documented. 

Ability  3 

Adequate  resources  and  funding  are  provided  for  managing  test  objectives. 

Activity  1 

The  software  test  group  and  the  software  engineering  group  review  the  test 
objectives  before  test  cases  are  designed  to  meet  those  objectives. 

Activity  2 

The  software  test  group  uses  the  software  requirements  and  the  test  objectives 
as  a  basis  for  test  plans,  test  work  products,  and  testing  activities. 

Measurement  1 

Measurements  are  made  and  used  to  determine  the  status  of  the  activities  for 
managing  test  objectives. 
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Verification  2  The  activities  for  managing  the  test  objectives  are  reviewed  with  the  test 
manager  and  project  manager  on  both  a  periodic  and  event-driven  basis. 

5.3. 1 .3  Test  Planning/Tracking 

The  Software  Project  Planning  and  Software  Project  Tracking  and  Oversight  KPAs  in  the 
Repeatable  Maturity  Level  do  not  address  software  test  planning,  tracking,  and  oversight  specifically 
enough  because  the  test  objectives,  test  responsibilities,  and  test  work  products  are  not  discussed. 
Examples  of  software  work  products  at  the  Repeatable  Maturity  Level  do  not  include  test  work 
products.  The  only  test  plan  mentioned  at  Level  2  is  the  acceptance  test  plan  required  by 
subcontractors. 

The  CMM  basically  treats  test  planning  as  a  Level  3  engineering  activity.  Effective 
management  at  the  Repeatable  Maturity  Level  requires  fundamental  test  planning,  tracking,  and 
oversight  practices  that  address  test  objectives,  test  work  products,  and  test  activities  [GELP94].  We 
need  more  visibility  into  the  test  process  at  the  Repeatable  Maturity  Level.  For  test  development,  we 
need  that  same  process  visibility  required  at  the  conclusion  of  each  major  software  development 
phase.  You  can't  manage  what  you  don't  see.  Note  that  one  of  the  major  objectives  of  software 
quality  assurance  is  to  make  sure  that  you  are  performing  as  planned.  When  testing  is  not  planned 
adequately,  the  controls  are  not  in  place  for  SQA  and  management  to  evaluate  conformance  and 
progress. 

The  CMM  and  the  DOD-STD-2167A  prescribe  early  software  project  planning.  However, 
many  organizations  do  not  take  full  advantage  of  early  test  planning  and  test  development  practices 
[PAUL93]  [216788].  Testing  is  still  often  regarded  as  a  necessary  evil,  and  detailed  planning  efforts 
are  delayed  until  code  is  available  for  testing.  Brooks  told  us  long  ago  that  "testing  is  the  most  mis- 
scheduled  part  of  development"  [BR0075].  As  mentioned  earlier,  test  development  has  a  lifecycle. 
Recognition  of  that  lifecycle  will  go  a  long  way  to  improving  our  capabilities  for  estimating  and 
scheduling  test  activities.  Brooks  also  said,  "Developers  don't  remember  that  they  don't  understand  - 
they ...  happily  invent  their  way  through  the  gaps  and  obscurities"  [BR0075].  This  certainly  applies 
to  test  development.  The  goal  should  be  to  remove  as  many  test  development  activities  from  the  test 
execution  window  as  possible.  These  are  test  management  issues  that  need  to  be  addressed  early  in 
a  process  improvement  program  by  Level  1  organizations. 
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Some  example  recommended  Software  Test  Management  key  practices  for  test  planning 
(derived  from  the  Software  Project  Planning  KPA)  include 

Commitment  1  A  software  test  manager  is  designated  to  be  responsible  for  negotiating  testing 
commitments  and  developing  the  software  test  plan. 

Ability  2  Responsibilities  for  developing  the  software  test  plan  are  assigned. 

Ability  3  Adequate  resources  and  funding  are  provided  for  planning  the  test  activities. 

Activity  1  The  software  test  group  participates  on  the  project  proposal  team. 

Activity  5  A  test  development  lifecycle  is  defined  and  coordinated  with  the  software 
lifecycle. 

Activity  6  The  project's  test  plan  is  developed  according  to  a  documented  procedure. 

Activity  7  The  test  plan  is  documented. 

Activity  12  The  testing  schedule  is  derived  according  to  a  documented  procedure. 

Activity  13  Software  testing  focuses  on  software  risks  associated  with  cost,  schedule,  and 
technical  aspects  of  the  project. 

Measurement  1  Measurements  are  made  and  used  to  determine  the  status  of  the  test  planning 
effort. 

Some  example  recommended  Software  Test  Management  key  practices  for  tracking  and  oversight 
of  testing  (derived  from  the  Software  Project  Tracking  and  Oversight  KPA)  include 

Commitment  2  The  project  follows  a  written  organizational  policy  for  managing  the  test  effort. 

Ability  2  The  test  manager  explicitly  assigns  responsibility  for  test  work  products  and 

activities. 

Activity  8  The  project's  test  schedule  is  tracked,  and  corrective  actions  are  taken  as 
necessary. 

Activity  9  Test  engineering  technical  activities  are  tracked,  and  corrective  actions  are  taken 
as  necessary. 

Activity  12  The  test  group  conducts  periodic  internal  reviews  to  track  technical  progress, 
plans,  performance,  and  issues  against  the  test  plan. 
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5.3. 1 .4  Regression  Testing 

The  requirement  for  regression  testing  in  the  Software  Configuration  Management  KPA  is 
optional.  This  KPA  states  that  "reviews  and/or  regression  testing  are  performed  to  ensure  that 
changes  have  not  caused  unintended  effects  on  the  baseline"  [PAUL93]. 

Though  the  intent  of  the  Repeatable  Maturity  Level  deals  with  being  able  to  repeat  the  same 
basic  process  for  other  projects  with  the  same  likelihood  of  success,  the  very  concept  of  repeatability 
is  in  question  with  respect  to  the  project.  The  importance  of  regression  testing  is  not  adequately 
addressed  at  Level  2.  The  Software  Configuration  Management  guidelines  for  regression  testing  and 
configuration  control  of  test  work  products  may  need  to  be  strengthened  after  changes  are  made. 
This  will  improve  our  capabilities  to  repeat  our  process  on  current  and  subsequent  projects  and  will 
better  support  reuse  of  appropriate  test  work  products. 

5 . 3 . 1 . 5  Timing  of  T est  Process  Improvements 

Because  testing  is  treated  as  a  technical  activity,  important  test  management  issues  may  not 
surface  as  high-priority  problem  areas  when  CB  A  IPIs  (or  SPAs)  are  conducted  using  the  CMM  as 
a  guide.  Phil  Koltun,  of  Harris  Corporation,  wrote,  "The  SEI's  CMM  postpones  treatment  of 
software  testing  until  Level  3.  That's  unfortunate  because  it  discourages  less  mature  organizations 
from  devoting  process  improvement  attention  to  this  vital  activity"  [KOLT93]. 

Unless  we  consider  software  test  management  practices  early  in  our  process  improvement 
efforts,  we  may  be  headed  for  the  same  kinds  of  problems  that  we  experienced  in  the  past  as  we 
involved  unprepared  testers  late  in  our  development  projects.  Effective  testing  practices  improve 
overall  development  effectiveness.  A  Software  Test  Management  KPA  will  give  added  emphasis  to 
testing  to  improve  software  development  capabilities. 

5.3.2  Test  Engineering  Concerns 

The  role  of  software  testing  has  evolved  in  the  industry  over  the  last  several  decades  from 
demonstration,  to  detection  of  errors,  to  evaluation,  and  finally  to  defect  prevention  [GELP94]. 
Unfortunately,  testing's  image  has  remained  poor  in  many  organizations  because  of  excessive  costs 
and  ineffective  practices.  An  example  of  this  poor  image  is  reflected  in  the  CMM's  statement, 
"During  a  crisis,  projects  typically  abandon  planned  procedures  and  revert  to  coding  and  testing" 
[PAUL93].  Part  of  this  poor  image  has  resulted  from  testing  being  burdened  with  having  to  include 
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debugging  and  rework  costs.  Test  costs  should  not  include  debugging  and  rework.  Perhaps  the 
CMM  should  say, "...  coding,  testing,  and  rework." 

The  application  of  effective  test  development  practices  (a  CMM  Level  3  issue)  and 
appropriate  levels  of  automation  of  test  practices  (management  and  engineering)  will  reduce  costs 
and  improve  the  quality  of  our  software.  The  following  subject  areas  describe  some  CMM  test 
engineering  concerns  and  recommendations  for  enhancing  the  CMM  to  better  address  effective  test 
engineering  practices. 

5.3.2. 1  Test  as  a  Process  Improvement  Mechanism 

Testing  was  not  specifically  mentioned  as  a  process  improvement  mechanism  in  the  CMM  at 
any  CMM  level.  The  example  software-related  defect  prevention  groups  at  CMM  Level  5  did  not 
include  the  software  test  group. 

Many  organizations  believe  that  you  cannot  test  quality  into  software.  While  this  is  true  after 
the  software  is  built,  it  is  not  true  during  development.  Test  development  activities  can  play  a 
significant  role  in  preventing  defects  during  early  development  phases  [GELP94].  In  fact,  building 
test  cases  from  requirements  may  be  the  most  objective  and  effective  mechanism  to  evaluate 
requirements  (and  determine  testability  -  a  Level  2  practice)  during  requirements  analysis  and  design. 
Lt  Col  Mark  Kindi  considers  testing  as  "the  most  important  method  for  producing  error  data 
necessary  to  guide  process  improvement"  [KIND92].  Software  testers  can  make  significant 
contributions  to  prevent  defects  and  should  be  consulted  with  respect  to  process  improvement.  Note 
that  test  process  improvement  must  always  include  an  increased  capability  to  measure  development 
progress. 

5. 3. 2.2  Independence  Versus  Interdependence 

The  discussion  about  the  independence  of  the  test  group  needs  to  be  enhanced  in  the  Defined 
Maturity  Level  to  address  an  interdependence  of  the  test  group  and  the  software  development  group. 
Independence  discourages  the  partnership  that  should  exist  between  developers  and  testers  in 
producing  quality  software.  It  also  discourages  early  involvement  of  testers  on  the  project. 
Admittedly,  evaluation  activities  conducted  by  testers  must  be  planned  and  conducted  separately  from 
development  activities  to  be  eflTective.  However,  there  are  significant  benefits  to  having  an 
interdependence  between  the  developers  and  testers  to  provide  a  common  understanding  of 
requirements  and  tests  [GELP94].  The  concern  about  the  developers  knowing  what  the  testers  will 
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be  testing  for  is  unfounded.  We  want  systems  that  have  passed  a  "comprehensive"  set  of  tests,  where 
comprehensiveness  is  based  on  an  adequate  consideration  and  coverage  of  known  concerns. 

5.3.2.3  The  Role  of  Automation 

Because  the  CMM  generally  focuses  on  management  concerns  at  Level  2,  some  people  say 
that  adopting  CASE  tools  is  a  Level  3  activity.  The  CMM  wisely  councils  that  "the  benefits  of  better 
methods  and  tools  cannot  be  realized  in  the  maelstrom  of  an  undisciplined,  chaotic  process" 
[PAUL93].  While  it  is  important  not  to  automate  chaos,  some  elements  of  the  software  development 
process,  when  understood  well  enough,  can  be  considered  for  automation  early  in  an  improvement 
effort.  The  CMM  also  states  that,  "Software  engineering  technical  activities  are  tracked  and 
corrective  actions  are  taken  as  necessary"  [PAUL93].  Correcting  involves  improvements  to  activities 
that  use  improved  practices  and  tools.  Effective  testing  demands  some  amount  of  automation  for 
systems  of  any  appreciable  size.  Michael  Deutsch  says  "the  evaluation  of  the  output  data,  if 
performed  manually,  is  likely  to  be  a  tedious,  time-consuming,  and  error-prone  process  for  all  but  the 
most  elementary  of  tests"  [DEUT82]. 

An  important  question  about  software  testing  isn't  whether  or  not  an  organization  should 
automate  but  rather,  "Are  there  technologies  and  tools  available  that  will  enable  organizations  to 
effectively  test  their  software  and  are  the  tools  economical  in  terms  of  (1)  speeding  up  the 
development  processes  and  (2)  maximizing  the  quality  of  all  product  releases  to  minimize  future  error 
correction  efforts?"  These  questions  need  to  be  asked  for  each  project.  Our  investigations  have 
found  that  there  are  a  number  of  software  test  tools  that  can  significantly  help  testers  on  a  number 
of  platforms  and  software  engineering  environments.  Many  of  these  tools  can  be  used  by 
organizations  with  a  Level  1  rating.  Note  that  several  Level  2  Ability  Common  Features  discuss 
providing  adequate  resources  and  finding,  which  includes  the  availability  of  appropriate  tool  support. 

Tool  adoption  requires  careful  planning,  evaluation,  and  trial  use.  However,  not  all  CASE 
tools®  are  implemented  alike  nor  do  they  require  the  same  level  of  maturity  to  adopt.  Tools  that  don't 
cause  developers  and  testers  to  have  to  rethink  their  problems  but  rather  help  them  better  view  their 
problems  are  good  candidate  tools  in  the  initial  stages  of  process  improvement.  Several  types  of 
testing  tools  can  be  considered  early  in  a  process  improvement  effort  and  can  reduce  resistance  to 
organizational  change.  Jerry  Durant  believes  that,  "Test  tools  should  provide  immediate  benefits" 


®  CASE  tools  include  tools  that  support  requirements  analysis,  design,  code,  test,  documentation,  etc.  [IEEE92]. 
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[DURA93].  Some  of  those  benefits  are  intangible  but  still  very  important,  such  as  an  increased  level 
of  confidence  in  testing  accuracy  and  comprehensiveness.  Some  example  types  of  test  tools  that 
encourage  process  improvement  are  discussed  in  the  next  several  paragraphs. 


a.  Defect  tracking  tools  can  help  manage  the  test  and  rework  effort  at  all  maturity  levels. 

b.  Static  analyzers  can  automate  many  code  review  tasks  (and  can  sometimes  give  you  more 
diagnostics  and  code  measurements  than  you  might  initially  care  for).  Many  static  analyzers 
are  considered  "no  brainers"  to  use  (as  simple  as  compilers)  and  can  be  adopted  at  all  maturity 
levels.  Note  that  sophisticated  code  measurement  systems  may  require  that  mature  practices 
be  in  place  before  adopting  them. 

c.  While  many  coverage  analyzers  have  found  their  way  to  a  dusty  shelf,  coverage  analysis  is 
fundamental  to  the  test  process  and  is  considered  "criminal"  not  to  know  how  much  code  was 
exercised  during  development  [BEIZ90].  Maybe  coverage  analyzers  should  be  considered 
at  Level  2.  However,  adopting  coverage  analyzers  will  require  some  organizational  changes. 
Note  that  coverage  analysis  was  promoted  to  a  CMM  Level  3  test  engineering  activity  from 
a  Level  4  measurement  activity  in  the  old  SEI  software  process  maturity  model  published  in 
1987  [HUMP87]. 

d.  Capture/replay  and  test  management  tools  automate  much  of  the  test  execution  process  and 
can  vastly  improve  regression  testing  capabilities.  But  a  significant  investment  may  be 
required  to  buy  and  learn  the  tools  before  benefits  can  be  derived.  Changes  may  require  a 
major  rework  of  the  test  scripts,  especially  in  graphical  user  environments.  However,  those 
changes  may  also  require  considerable  rethinking  in  a  manual  or  ad  hoc  testing  environment 
if  effective  testing  is  a  goal.  Note  that  there  are  no  savings  with  capture/replay  tools  during 
the  first  time  a  test  is  run  [GRAH93]. 

e.  Requirements-based  test  case  generators  provide  an  interesting  capability  that  automates 
some  test  development  activities  at  the  software  requirements  analysis  and  design  phases. 
Perhaps  the  greatest  benefit  that  these  kinds  of  tools  offer  is  the  promotion  of  gathering  and 
improving  requirements  information.  Test  case  generators  build  several  classes  of  test  cases 
(function,  logic,  boundary  value,  equivalence  partitions,  etc.)  from  available  requirements 
information.  These  tools  may  also  require  a  significant  investment  to  adopt  in  the 
organization,  but  the  benefits  can  be  very  impressive  [PATR92]. 

For  every  practice,  the  CMM  advocates  that  adequate  resources  and  funding  be  provided,  and 
appropriate  tools  be  made  available  to  accomplish  the  work.  SEPG-type  organizations  are  reminded 
not  to  forget  that  test  process  improvement  includes  an  appropriate  level  of  technology  and  tool 
automation  improvements.  Another  maturity-pull  effect  can  be  felt  with  the  adoption  of  appropriate 
Level  5  Technology  Change  Management  KPA  practices  when  adopting  tools. 
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5.4  Recommendations 

A  greater  appreciation  for  the  CMM  will  come  from  studying  it  and  applying  its  principles. 
Perhaps  the  best  way  to  understand  something  is  to  try  to  use  it  and  improve  it.  Every  organization 
should  carefully  consider  the  practices  advocated  by  the  CMM  and  adopt  those  that  make  sense  for 
them  in  a  planned,  orderly  progression. 

Many  of  the  same  management  practices  used  to  manage  software  development  can  be 
effectively  applied  toward  the  management  of  test  development.  Obtaining  management  commitment 
and  support  as  the  CMM  advises  is  fundamental  to  a  software  process  improvement  effort.  The 
CMM,  supplemented  with  the  proposed  Software  Test  Management  Key  Process  Area,  may  enhance 
your  organization's  understanding  of  their  testing  roles  and  may  advocate  to  your  management  and 
your  technical  staff  that  improved  testing  practices  should  be  considered  early  in  a  software  process 
improvement  effort. 

Effective  software  test  engineering  practices  plays  a  vital  role  in  improving  an  organization's 
overall  software  development  capabilities.  Early  involvement  of  testers  with  developers  helps  build 
quality  systems.  Finally,  you  don't  have  to  be  a  Level  3  organization  before  considering  appropriate 
automation  of  well-known  testing  activities. 

This  technology  report  provides  a  useful  forum  for  exchanging  ideas  between  DoD,  SEI,  and 
industry.  Because  of  the  attention  focused  on  the  CMM,  many  organizations  are  looking  to  it  to 
guide  their  process  improvement  efforts.  Contact  the  STSC  for  more  information  about  applying  the 
principles  of  effective  software  test  engineering  and  about  improving  the  material  in  these  reports. 
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Appendix  A.1:  Product  Sheets  by  Tool  Name 


Appendix  A.l:  Product  Sheets  by  Tool  Name 
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Appendix  AJ:  Product  Sheets  by  Tool  Name 
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Appendix  AJ:  Product  Sheets  by  Tool  Name 
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Appendix  A. 1 :  Product  Sheets  by  Tool  Name 
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Appendix  A. 1:  Product  Sheets  by  Tool  Name 
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Software  Blacksmiths,  Inc. 
Laura  Searle 

905-858-4466  Fax: 


905-858-4466 


6064  Saint  Ives  Way 
Mississauga,  ON  L5N-4M1  Canada 


Tool;  C-DOC  Vendor:  software  Blacksmiths,  Inc. 

Version:  5.1  Release  Date:  11/01/93  POC:  Laura  Searle 

Number  Sold:  5,000+  Single  User  Price:  $199-$299(150K-LOC)  Phone:  905-858-4466  Fax:  905-858-4466 

Report  Date:  3/03/93  Report  Updated:  6/30/94  E-mail: 

Address:  6064  Saint  Ives  Way 

Evaluation  Copy  Available  Site  License  Available  Mississauga,  ON  L5N-4M1  Canada 

Description:  C-DOC  is  a  set  of  tools  that  analyze  and  document  C  and  C++  programs.  It  produces  clear  documentation  in  text  files, 
and  restructures  code  including  insertion  of  self  documentation  comment  blocks.  Requires  no  source  code  changes.  Handles  up  to  1  OK 
lines. 

Classiflcation:  Coding,  Documentation,  Quality  Assurance,  Metrics,  Reengineering,  Complexity  Measurer,  Cross  Referencing  Tool, 
Defect/Change  Tracker,  Redocumenter,  Restructurer,  Size  Measurer,  Structure  Checker 
Features:  SIOC  Actuals 
Languages  Supported:  c,  C++ 

Configurations:  PC/MS-DOS,  PC/OS/2 


Release  Date:  1/01/92 
Single  User  Price:  $59 
Report  Updated:  6/30/94 


Tool:  C-METRIC  Vendor:  software  Blacksmiths,  Inc. 

Version:  5.1  Release  Date:  1/01/92  POC:  Laura  Searle 

Number  Sold:  5,000+  Single  User  Price:  $59  Phone:  905-858-4466  Fax:  905-858-4466 

Report  Date:  4/07/93  Report  Updated:  6/30/94  E-mail: 

Address:  6064  Saint  Ives  Way 

Evaluation  Copy  Available  Site  License  Available  Mississauga,  ON  L5N-4M1  Canada 

Description:  C-METRIC  analyzes  and  documents  the  "cyclomatic"  path  complexity  of  C  and  C++  programs,  counts  code  lines,  lines 
with  comments,  lines  with  code,  and  C  statements.  Part  of  the  C-DOC  tool  suite. 

Classification:  Coding,  Documentation,  Quality  Assurance,  Metrics,  Reengineering,  Complexity  Measurer,  Size  Measurer 
Features:  SLOC  Actuals 
Languages  Supported:  c,c++ 

Configurations:  MS-DOS,  PC/OS2  _ 


Software  Blacksmiths,  Inc. 

Laura  Searle 

905-858-4466  Fax:  905-858-4466 

6064  Saint  Ives  Way 
Mississauga,  ON  L5N-4M1  Canada 


Tool:  C-REF 

Version:  5.1 

Number  Sold:  5,000+  S 
Report  Date:  4/07/93 

Evaluation  Copy  Available 


Release  Date:  1 1/01/93 
Single  User  Price:  $59 
Report  Updated:  5/24/94 

e  Site  License  Available 


Vendor: 

POC: 

Phone: 

E-mail: 

Address: 


Software  Blacksmiths,  Inc. 

Laura  Searle 

905-858-4466  Fax:  905-858-4466 

6064  Saint  Ives  Way 
Mississauga,  ON  L5N-4M1  Canada 


Description:  C-REF  analyzes  C  and  C++  programs  and  produces  a  summary  or  detailed  cross-reference  of  identifiers  used  (LOCALS, 
GLOBALS,  PARAMETERS,  and  DEHNES).  Part  of  the  C-DOC  tool  suite. 

Classification:  Coding,  Documentation,  Quality  Assurance,  Reengineering,  Cross  Referencing  Tool 

Features: 

Languages  Supported:  c,  C++ 

Configurations:  MS-DOS,  PC/OS2  _ 


Tool:  c-scAPE 

Version:  4.0.1 
Number  Sold:  ~  I 

Report  Date:  3/03/93 

Evaluation  Copy  Available 


Release  Date:  1/01/93 
Single  User  Price:  $499 
Report  Updated:  12/03/93 
Newsletter  Available 
Site  License  Available 


Vendor: 

POC: 

Phone: 

E-mail: 

Address: 


LIANT  Software  Corp. 

April  Seaberg 

508-872-8700  x370  Fax: 

959  Concord  St. 

Framingham,  MA  01701 


508-626-2221 


Description:  C-SCAPE  is  an  object-oriented  interface  management  system.  The  C-SCAPE  library  is  a  collection  of  functions  for 
working  with  windows,  data  entry  screens,  graphical  images,  menus,  and  text  editing. 

Classification:  System  Simulation,  Design,  Coding,  Testing,  Reengineering,  Software  Engineering  Environment,  Cross  Referencing 
Tool,  Debugger,  Reverse  Engineering 

Features: 

Languages  Supported:  C,  C++ 

Configurations:  UNIX,  Intel  (386/486),  and  Sun  SPARC _ 
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Tool  I  C- Vision  for  C 

Vendor:  Cimpel  software 

Version:  3.1  Release  Date:  5/01/93 

POC:  AnnaGimpel 

Number  Sold:  1,000  Single  User  Price:  $139 

Phone:  215-584-4261  Fax:  215-584-4266 

Report  Date:  12/27/93  Report  Updated:  5/16/94 

E-mail: 

Address:  3207  Hogarth  Ln. 

Site  License  Available 

Collegeville,  PA  19426 

Description:  Generates  a  cross-reference,  hierarchy  tree  of  C 

source  code,  also  included  an  outlined  listed  and  a  reformatter. 

Classification:  Coding,  Documentation,  Reengineering,  Cross  Referencing  Tool,  Reformatter,  Stmcture  Checker  1 

Features: 

Languages  Supported:  C 

Configurations:  PC/MS-DOS,  PC/OS/2 

Release  Date: 

Single  User  Price:  $250.00 
Report  Updated:  5/18/94 


Cater  Software 
Tech.  Support 
503-581-5622 


Fax:  503-362-2318 


525  Ferry  St.,  SE,  Suite304C 
Salem,  OR  97301 


Tool:  c/ANALYST  Vcndor:  Cater  Software 

Version:  2.12  Release  Date:  POC:  Tech.  Support 

Number  Sold:  1,000  Single  User  Price:  $250.00  Phone:  503-581-5622  Fax:  503-362-2318 

Report  Date:  8/24/93  Report  Updated:  5/18/94  E-mail: 

Address:  525  Ferry  St.,  SE,  Suite304C 
Site  License  Available  Salem,  OR  97301 

Description:  C/ANALYST  generates  documentation  to  help  a  programmer  understand  a  program.  It  uses  static  source  code  analysis  to 
create  a  database  representing  the  entire  program.  From  this  database  a  variety  of  listings  can  be  generated. 

Classification:  Coding,  Documentation,  Reengineering,  Cross  Referencing  Tool,  Data  Reducer  &  Analyzer,  Syntax  &  Semantics 
Analyzer 

Features: 

Languages  Supported:  C 

Configurations:  PC/MS-DOS _  _ 


Number  Sold:  25+ 
Report  Date:  1/04/93 
Training  Available 


Single  User  Price:  $13,000-$23,000 
Report  Updated:  6/30/94 
Newsletter  Available 


TooIj  C3Ada  Symbolic  Debugger  VBIldorj  Concurrent  Computer  Corp. 

Version:  2.4  Release  Date:  2/01/91  POC:  Linda  G.  Lewis 

Number  Sold:  25+  Single  User  Price:  $13,000-$23,000  Phone:  908-758-7000  Fax:  908-630-4833 

Report  Date:  1/04/93  Report  Updated:  6/30/94  E-mail: 

Training  Available  Newsletter  Available  Address:  2  Crescent  PI. 

oceanport,  NJ  07757 

Description:  C3Ada  Symbolic  Debugger  enables  the  user  to  locate  faults  by  providing  places  to  insert  break-points,  examining  and 
modifying  the  values  of  objects  and  more.  Operates  in  a  CASEmate  integration  framework  environment  to  provide  a  single 
environment  for  software  development. 

Classification:  Coding,  Testing,  Configuration  Management,  Compiler,  Cross  Referencing  Tool,  Debugger,  Library,  Linker,  Syntax 
&  Semantics  Analyzer 

Features: 

Languages  Supported:  Ada 

Configurations:  Series  3200-OS/32  R9.1 


Tool:  CA-COBOLVISION/Analy*er  Vcodor: 

Version:  1,0  Release  Date:  2/01/93  POC: 

Number  Sold:  -  Single  User  Price:  (Contact  Vendor)  Phone: 

Report  Date:  3/02/93  Report  Updated:  6/30/94  E-mail: 

Address: 


Computer  Associates  Inti. 
Alan  Brown 


703-709-4740 


Fax:  703-709-4820 


Address:  12120  Sunset  Hills  Rd. 

Reston,  VA  22090 

Description:  CA-COBOLVISION/Analyzer  lengthens  the  power  of  Computer  Associates  mainframe  products  in  conjunction  with 
COBOL-Intelligent  Visualization  and  Navigation. 

Classification:  Coding,  Reengineering,  Cross  Referencing  Tool,  Structure  Checker 

Features: 

Languages  Supported:  COBOL 
Configurations:  IBM/MVS,  IBM/VSE,  PC/MS-DOS 
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Tool:  CheckItLAN 

Version:  2.1 
Number  Sold:  1,000+ 
Report  Date:  3/01/93 
Training  Available 
Evaluation  Copy  Available 


Vendor: 

Release  Date:  3/01/92  POC: 

Single  User  Price:  (Contact  Vendor)  Phone: 

Report  Updated:  6/30/94  E-mail: 

Address: 

Site  License  Available 


Touchstone  Software  Corp. 
Technical  Support 
800-531-0450  Fax: 

2130  Main  ST.  Suite  250 
Huntington  Beach,  CA  92648 


714-960-1886 


Description:  Checkit  LAN  lets  you  troubleshoot  problems  on  a  Novell  LAN  from  your  PC.  It  allows  the  user  to  perform  a  software 
inventory,  scan  every  PC  for  viruses,  and  monitor  the  network  for  problems  and  usage. 

Classification:  Testing,  Data  Extractor,  Data  Reducer  &  Analyzer,  Network  Analyzer,  Run-Time  Error  Checker 

Features: 

Languages  Supported:  - 

Configurations:  PC _ _ 


Tool:  Checkit  PRO:AnaIyst 

Version:  1.0  Release  Date:  11/01/93 

Number  Sold:  900  Single  User  Price:  $149 
Report  Date:  11/30/93  Report  Updated:  6/30/94 


V  endor :  Touchstone  Software  Corp. 

POC:  Scott  Mackay 

Phone:  800-531-0450  Fax:  714-960-1886 

E-mail:  70762, 2663(cserve) 

Address:  2130  Main  ST.  Suite  250 

Huntington  Beach,  CA  92648 

Description:  Checkit  PRO: Analyst  analyzes  system  performances,  reviews  software  installation  requirements,  and  avoids  system 
set-up  conflicts  when  installing  hardware.  Also  assists  with  hardware  and  software  purchase  decisions,  installing  upgrades,  or 
determining  if  repairs  are  required. 

Classification:  Requirements  Analysis,  Testing,  Configuration  Management,  Quality  Assurance,  Performance/Timing  Analyzer 

Features: 

Languages  Supported:  -- 

Configurations:  OS/2,  Windows,  PC/MS-DOS _ 


TooL  Checkpoint 

Version:  2.1 
Number  Sold:  - 
Report  Date:  2/17/93 


Vendor: 

POC: 
Phone: 
E-mail: 
Address: 


Software  Productivity  Research,  Inc. 

Lynne  Caramanica 

617-273-0140  Fax:  617-273-5176 


Release  Date: 

Single  User  Price:  (Contact  Vendor) 

Report  Updated:  5/24/94 

1  New  England  Executive  Park 
Burlington,  MA  01803-5005 

Description:  Checkpoint  is  an  applied  software  measurement  tool  that  enables  a  precise  estimate,  measurement,  and  assessment  of 
software  project  variables.  Uses  SPR's  methodology  to  measure  all  the  factors  that  influence  the  development  of  software. 
Classification:  Quality  Assurance,  Metrics,  Size  Measurer 
Features:  SLOC  Actuals 
Languages  Supported:  All 
Configurations:  PC,  UNIX _ 


Tool:  CLAS  2000 

Version:  - 
Number  Sold:  - 
Report  Date:  1/04/93 


Vendor: 

POC: 

(Contact  Vendor)  Phone: 

E-mail: 
Address: 


Biomation 

Gregory  A.  Richardson 
800-835-5996 


Release  Date: 

Single  User  Price:  (Contact  Vendor)  Phone:  800-835-5996  Fax:  - 

Report  Updated:  6/30/94 

3875  Thundercloud  Dr 
Colorado  Springs,  CO  80920 

Description:  CLAS  2000  is  a  logic  analysis  system  that  performs  a  number  of  tests  on  hardware,  including  timing  speed,  memory,  and 
glitch  counting. 

Classification:  Testing,  Performance/riming  Analyzer,  Run-Time  Error  Checker 

Features: 

Languages  Supported:  - 

Configurations:  HP/HP-ux,  PCAVindow  3.0,  Rise  6000 _ 
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Tool:  CodeCenter  V  CllClOri  CenterLine  Software,  Inc. 

Version:  -  Release  Date:  POC:  DickBurgett 

Number  Sold:  -  Single  User  Price:  (Contact  Vendor)  Phone:  703-749-1100  Fax:  703-749-1108 

Report  Date:  1/27/93  Report  Updated:  6/30/94  E-mail:  burgetti@centerline.com 

Address:  7926  Jones  Branch  Dr.,  Suite  1000 
McLean,  VA  22102 

Description:  CodeCenter  is  an  integrated  C  programming  environment  that  combines  an  interactive  interpreter,  a  source-level 
debugger,  graphic  browsers,  and  an  incremental  linker-loader. 

Classification:  Coding,  Testing,  Compiler,  Cross  Referencing  Tool,  Debugger,  Interpreter,  Linker 

Features: 

Languages  Supported:  c 

Configurations:  DEC,  HP,  IBM,  Sun,  UNIX  _ 


Vendor: 

Release  Date:  POC; 

Single  User  Price:  (Contact  Vendor)  Phone: 

Report  Updated:  6/30/94  E-mail: 

Address: 


Release  Date:  1/01/88 
Single  User  Price:  $495 
Report  Updated:  6/30/94 


ABRAXAS  Software,  Inc. 
Elizabeth  Layton 
503-244-5253  Fi 


Fax:  503-244-8375 


5530  S.W.  Kelly 
Portland,  OR  97201 


Tool:  CodeCheck  Vendor:  abraxas  software,  Inc. 

Version:  5.0  Release  Date:  1/01/88  POC:  Elizabeth  Layton 

Number  Sold:  600  Single  User  Price:  $495  Phone:  503-244-5253  Fax:  503-244-8375 

Report  Date:  3/09/93  Report  Updated:  6/30/94  E-mail: 

Address:  5530  S.W.  Kelly 

Evaluation  Copy  Available  Site  License  Available  Portland,  OR  97201 

Description:  CodeCheck  analyzes  C  and  C++  source  code.  CodeCheck  is  designed  to  enhance  the  effectiveness  and  efficiency  of 
project  management  by  analyzing  the  portability,  maintainability,  and  style  of  the  source  code. 

Classification:  Coding,  Project  Management,  Quality  Assurance,  Reengineering,  Auditor,  Code  Change  Monitor,  Complexity 
Measurer,  Reverse  Engineering,  Size  Measurer,  Syntax  &  Semantics  Analyzer 
Features:  SLOC  Actuals 
Languages  Supported:  C,  C++ 

Configurations:  MS-DOS,  OS2,  UNIX,  Mac/Mac  OS,  PC/DR-DOS,  VAX/VMS 


Tool:  COHESION  Team/SEE  Vendor: 

Version:  1.0  Release  Date:  1/01/94  POC: 

Number  Sold:  Single  User  Price:  (Contact  Vendor)  Phone: 

Report  Date:  1/27/93  Report  Updated:  6/30/94  E-mail: 

Address: 


Digital  Equipment  Corp. 

Technical  Support 
800-344-4825  Fax: 


603-881-2381 


Address:  146  Main*  St. 

Maynard,  MA  01754-2571 

Description:  COHESION  Team/SEE  for  OSF/1 AXP  is  a  set  of  software  tools  for  managing  data  and  development  processes  for 
medium-to-large-scale  UNIX  software  engineering  projects  distributed  across  multivendor,  multisite  environments.  It  provides  the  base 
environment  to  support  the  fuU  software  development  lifecycle. 

Classification:  Coding,  Testing,  Configuration  Management,  Quality  Assurance,  Metrics,  Database,  Software  Engineering 
Environment,  Data  Extractor,  Debugger,  Defect/Change  Tracker 

Features: 

Languages  Supported:  Ada,  C++ 

Configurations:  DECstation/OSF/l  _ 


Tool:  COMPARE 

Vendor: 

Aldon  Computer  Group 

Version:  - 

Release  Date: 

POC: 

Technical  Support 

Number  Sold:  ~ 

Single  User  Price:  (Contact  Vendor) 

Phone: 

510-839-3535  Fax:  510-839-2894 

Report  Date:  1/04/93 

Report  Updated: 

E-mail: 

Address: 

401  15th  St. 

Oakland,  CA  94612 

Description;  COMPARE  compares  files  to  determine  differences 
ClassiHcation:  Testing,  Comparator 

Features: 

Languages  Supported:  All 

ConHgurations:  HP-3000/MPE 
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Tool:  DEC  FUSE  Call  Graph  Browser  Vendor:  Digital  Equipment  Corp. 

Version:  1.2  Release  Date:  2/01/93  POC:  Technical  Support 

Number  Sold:  -  Single  User  Price:  (Contact  Vendor)  Phone:  800-344-4825  Fax:  603-881-2381 

Report  Date:  1/04/93  Report  Updated:  5/16/94  E-mail: 

Address:  146  Main*  St. 

Maynard,  MA  01754-2571 

Description:  DEC  FUSE  Call  Graph  Browser  is  a  structure  checker. 

Classiflcation:  Reengineering,  Structure  Checker 

Features: 

Languages  Supported:  Ada,  c,  C++,  fortran,  Pascal 
Configurations:  DECstation/ULTRIX,  Sun 


Tool:  DEC  FUSE  Cross-Reference  V Glldori  Digital  Equipment  Corp. 

Version:  1.2  Release  Date:  2/01/93  POC:  Technical  Support 

Number  Sold:  -  Single  User  Price:  (Contact  Vendor)  Phone:  800-344-4825  Fax:  603-881-2381 

Report  Date:  1/04/93  Report  Updated:  5/16/94  E-mail: 

Address:  146  Main  St. 

Maynard,  MA  01754-2571 

Description:  DEC  FUSE  is  a  Cross  Referencing  Tool. 

Classification:  Coding,  Cross  Referencing  Tool 

Features: 

Languages  Supported:  Ada,  c,  C++,  FORTRAN,  Pascal 
Configurations:  DECstation/ULTRIX,  Sun 


Tool:  DEC/Test Manager  (DIM)  Vendor:  Digital  Equipment  Corp. 

Version:  11.2  Release  Date:  11/01/93  POC:  Technical  Support 

Number  Sold:  ~  Single  User  Price:  (Contact  Vendor)  Phone:  800-344-4825  Fax:  603-881-2381 

Report  Date:  1/05/93  Report  Updated:  6/30/94  E-mail: 

Address:  146  Mairr  St. 

Maynard,  MA  01754-2571 

Description:  DEC/Test  Manager  automates  regression  testing,  DTM  runs  user-supplied  tests,  and  the  results  are  automatically 
compared  to  their  expected  results.  DTM  operates  in  the  interactive  and  batch  modes  and  supports  the  DEC  Windows  environment. 
Classification:  Coding,  Testing,  Quality  Assurance,  Capture-Replay  Tool,  Test  Execution  Manager 

Features: 

Languages  Supported:  All 

Configurations:  Alpha  AXP/Open  VMS,  VAX/Open  VMS 


Tool:  DecisionVision  1  (DVl)  Vendor:  Software  Business  Management,  Inc. 

Version:  2.2  Release  Date:  12/01/93  POC:  Technical  Support 

Number  Sold:  ~  Single  User  Price:  (Contact  Vendor)  Phone:  508-692-4145  Fax:  508-692-7151 

Report  Date:  4/01/94  Report  Updated:  6/30/94  E-mail: 

Address:  234  Littleton  Rd.,  Suite  2E 
Westford,  MA  01886 

Description:  DecisionVision  1  (DVl)  collects  and  reports  measurement  information  for  managing  the  quality,  cost,  and  schedules  of 
software  development  projects.  It  captures  code  statistics  at  preset  intervals  for  calculating,  graphing  and  reporting  volatility  (code 
changes),  reliability  and  complexity  information. 

Classification:  Coding,  Testing,  Quality  Assurance,  Metrics,  Reengineering,  Complexity  Measurer,  Defect/Change  Tracker, 
Reliability  Analyzer,  Reverse  Engineering,  Size  Measurer,  Structure  Checker 

Features:  Automatic  Data  Collection,  SLOC  Actuals 
Languages  Supported:  Ada,  c,  COBOL,  FORTRAN 

Configurations:  Sun/Sun  OS _ 
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Clarity  Concepts  Systems 
Jerry  Sitner 

212-254-3358  Fax: 

14  Washington  PI. 

New  York,  NY  10003 


Tool:  Enforcer  n  Vendor:  Clarity  Concepts  Systems 

Version:  6  Release  Date:  1/01/84  POC:  Jerry  Sitner 

Number  Sold:  -  Single  User  Price:  (Contact  Vendor)  Phone:  212-254-3358  Fax:  -- 

Report  Date:  1/27/93  Report  Updated:  7/01/94  E-mail: 

Address:  14  Washington  PI. 

Evaluation  Copy  Available  Site  License  Available  New  York,  NY  10003 

Description:  Enforcer  n  is  a  quality  assurance  COBOL  n  programming  standards  monitor.  It’s  main  purpose  is  to  support  the  "clarity 
coding"  concept.  It  can  be  used  to  assure  clear  understandable  COBOL  coding  and  reduces  maintenance  cost. 

Classification:  Coding,  Documentation,  Configuration  Management,  Quality  Assurance,  Metrics,  Reengineering,  Auditor,  Code 
Change  Monitor,  Complexity  Measurer,  Size  Measurer,  Structure  Checker 
Features:  SLOC  Actuals 
Languages  Supported:  COBOL  n 

Configurations:  MVS,  VSE,  IBM  _ 


Tool:  Ensemble  Vcndor:  Cadre  Technologies,  Inc. 

Version:  -  Release  Date:  POC:  JeffJenks 

Number  Sold:  -  Single  User  Price:  (Contact  Vendor)  Phone:  408-562-0106  Fax:  408-727-1163 

Report  Date:  4/06/93  Report  Updated:  5/16/94  E-mail: 

Address:  2880  Lakeside  Dr.,  Suite  231 
Santa  Clara,  CA  95054 

Description:  Ensemble  is  a  collection  of  forward  and  reverse  engineering  tools.  It  captures  design  information  and  stores  it  in  its 
design  database,  documents  the  design,  generates  test  cases,  and  analyzes  test  completeness,  among  other  capabilities. 
Classification:  Design,  Coding,  Testing,  Quality  Assurance,  Reengineering,  Complexity  Measurer,  Coverage/Frequency  Analyzer, 
Cross  Referencing  Tool,  Forward  Engineering,  Requirements-Based  Test  Case  Generator,  Reverse  Engineering,  Structure  Checker 

Features: 

Languages  Supported:  C 

Configurations:  AIX,  Sun  OS,  ULTRIX _ 


518-387-6928 


Tool:  Environment  for  Code  Reengineering  (ENCORE)  V 611  dor:  General  Electric  Co.  (R  &  D) 

Version:  2.1  Release  Date:  6/01/92  POC:  Joel  N,  Sturman 

Number  Sold:  —  Single  User  Price:  $Contact  Vendor  Phone:  518-387-5457  Fax:  518-387-6928 

Report  Date:  1/28/93  Report  Updated:  7/01/94  E-mail: 

Address:  Bldg  K-1  Room  3C1,  PO  Box  8 
Site  License  Available  Schenectady,  NY  12301 

Description:  ENCORE  is  a  graphical  environment  that  supports  code  Reengineering  tasks  including  code  translation  from  FORTRAN 
to  Ada,  control  restructuring,  data  restructuring,  and  repackaging.  It  is  available  as  a  result  of  a  translation  service. 

Classification:  Coding,  Reengineering,  Reuse,  Software  Engineering  Environment,  Complexity  Measurer,  Data  Reengineering, 
Restructurer,  Source  Code  Translator 
Features: 

Languages  Supported:  CMS-2,  FORTRAN,  jovial 

Configurations:  Sun  3,  Sun  4  (Sparc  1+,  Sparc2,  etc.)  _ 


Tool:  ES  RE/Vision 
Version:  2.4 

Number  Sold:  -  i 

Report  Date:  4/08/93 
Training  Available 
Evaluation  Copy  Available 


Vendor; 

Release  Date;  1/01/92  POC: 

Single  User  Price:  (Contact  Vendor)  Phone: 

Report  Updated:  7/01/94  E-mail: 

Newsletter  Available  Address: 

Site  License  Available 


Eden  Systems  Corp. 

Melisa  Duffy 

800-288-9510  Fax:  317-843-2271 

9302  N.  Meridian  St.,  Suite  350 
Indianapolis,  IN  46260-1820 


Description:  ES  RE/Vision  combined,  Q/AUDITOR  and  Q/ARTISAN  let  the  user  evaluate,  improve  and  accept  COBOL  programs. 
The  COBOL  portfolio  is  measured  and  kept  and  rewritten  selectively  with  the  senior  programmer's  skill. 

Classification:  Coding,  Documentation,  Project  Management,  Quality  Assurance,  Metrics,  Reengineering,  Software  Engineering 
Environment,  Auditor,  Complexity  Measurer,  Reliability  Analyzer,  Size  Measurer,  Structure  Checker 
Features:  SLOC  Actuals 
Languages  Supported:  COBOL 
Configurations:  IBM/MVS,  PC 
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Array  Systems  Computing  Products,  Inc. 
Leonard  Vanek 

416-736-0900  Fax:  416-736-4715 

401  Magnetic  Dr.,  Units  24-26 
Downsview,  ON  M3J-3H9  Canada 


Tool:  Expert  Debugging  Software  Assistant  (EDSA)  V endOF :  Array  Systems  Computing  Products,  Inc. 

Version:  2.0  Release  Date:  8/01/91  POC:  Leonard  Vanek 

Number  Sold:  25+  Single  User  Price:  $3,750  Phone:  416-736-0900  Fax:  416-736-4715 

Report  Date:  1/26/93  Report  Updated:  E-mail: 

Address:  401  Magnetic  Dr.,  Units  24-26 

Evaluation  Copy  Available  Downsview,  ON  M3J-3H9  Canada 

Description:  EDSA  is  an  intelligent  assistant  that  analyzes  code  and  transmits  its  knowledge  to  you.  It  has  control  and  data  flow 
browsing,  correctness  verification  and  search  management. 

Classification:  Coding,  Testing,  Quality  Assurance,  Reengineering,  Debugger,  Reverse  Engineering,  Structure  Checker,  Syntax  & 
Semantics  Analyzer 

Features: 

Languages  Supported:  Ada 

Configurations:  UNIX,  MS-DOS,  Sun/SunOS,  VAX/VMS 


Tool:  Express  960 

Version:  Release  Date:  6/01/92  POC:  Sales 

Number  Sold:  200  Single  User  Price:  (Contact  Vendor)  Phone:  408-733-7837  Fax:  408-773-1073 

Report  Date:  2/04/93  Report  Updated:  7/01/94  E-mail: 

Training  Available  Address:  Argues  Ave.,  P.O.  Box  3166 

Site  License  Available  Sunnyvale,  CA  9408 8-3166 

Description:  The  Express-i960Cx  is  a  hardware/software  debugging  environment  for  i960  RISC  processors  that  can  be  purchased  with 
source-level  debugger  or  compiler  tools  for  a  complete  development  environment. 

Classiflcation:  Coding,  Testing,  Quality  Assurance,  Reengineering,  Software  Engineering  Environment,  Assembler,  Compiler,  Cross 
Referencing  Tool,  Data  Reducer  &  Analyzer,  Debugger,  Performance/Timing  Analyzer,  Test  Instrumenter 
Features: 

Languages  Supported:  Assembler,  c,  C++ 

Configurations:  PC,  Sun _ 


V endor:  step  Engineering 


Fax:  408-773-1073 


Tool:  F-SCAN 
Version:  1.1 
Number  Sold:  15+ 
Report  Date:  1/26/93 
Training  Available 


Release  Date:  6/01/87 
Single  User  Price:  $4,950 
Report  Updated: 

Newsletter  Available 


Vendor: 

POC: 

Phone: 

E-mail: 

Address: 


Inti.  Logic  Corp. 

Technical  Support 
415-989-7223  I 

100  Pine  St.,  Suite  2760 
San  Francisco,  CA  94111 


415-989-0252 


Evaluation  Copy  Available  San  Francisco,  CA  94111 

Description:  F-SCAN  is  a  static  analyzer  of  FORTRAN  source  programs  that  provides  call  trees.  Cross  Referencing  Tools,  and 
analysis  of  code  interfaces. 

Classification:  Coding,  Requirements  Analysis,  Documentation,  Configuration  Management,  Quality  Assurance,  Reengineering, 
Cross  Referencing  Tool,  Redocumenter,  Structure  Checker,  Syntax  &  Semantics  Analyzer 

Features: 

Languages  Supported:  FORTRAN,  PL/l 
Configurations:  DEC,  DG,  IBM,  Prime 


Tool:  FCount  Vendor:  SAIC-Dayton 

Version:  Release  Date:  POC:  Debbie  Dyer 

Number  Sold:  -  Single  User  Price:  (Contact  Vendor)  Phone:  619-535-7652  Fax:  619-546-6833 

Report  Date:  3/31/94  Report  Updated:  7/01/94  E-mail; 

Address:  10260  Campus  Point  Dr.,  M/S  12 
San  Diego,  CA  92121 

Description:  FCount  line  counter  available  as  Ada  source  code,  and  counts  logical  FORTRAN  lines  of  code. 

Classification:  Quality  Assurance,  Metrics,  Size  Measurer 
Features:  SLOC  Actuals 
Languages  Supported:  FORTRAN 

Configurations:  Mac,  PC/MS-DOS,  UNIX  _ 
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Tiburon  Systems,  Inc. 
Gregory  M.  Pope 
408-371-9400 

2085  Hamilton  Ave. 
San  Jose,  CA  95125 


Fax:  408-371-2518 


Tool:  FERRETT  Vendor:  Tiburon  Systems,  Inc. 

Version:  13,3  Release  Date:  12/01/91  POC:  Gregory  M.  Pope 

Number  Sold:  10+  Single  User  Price:  (Contact  Vendor)  Phone:  408-371-9400  Fax:  408-371-2518 

Report  Date:  1/07/93  Report  Updated:  7/01/94  E-mail: 

Address:  2085  Hamilton  Ave. 

Evaluation  Copy  Available  San  Jose,  CA  95125 

Description:  FERRET  is  a  Computer-Aided  Software  Test  (CAST)  workstation.  It  automates  test  design  and  regression  testing,  is 
nonintrusive,  and  works  with  most  languages,  OSs,  windows,  and  hardware. 

Classification:  Requirements  Trace,  Design,  Coding,  Testing,  Quality  Assurance,  Reuse,  Capture-Replay  Tool,  Status  Displayer,  Test 
Execution  Manager  , 

Features:  GUI-based  Testing,  Text-based  Testing 

Languages  Supported:  All 

Configurations:  Apollo,  Harris,  Honeywell,  HP,  IBM,  Mac,  PC/MS-DOS,  PC/Window  3.0,  Plexus,  Prime,  Pyramid,  Rainbow, 
Rational,  Sequent,  SGraphics,  Sperry,  Sun,  UNIX  _ 


Tool:  File  Edit  utility  (FEU)  Vendor:  Applied  Logic  Corp. 

Version:  2.0  Release  Date:  1/01/84  POC:  Technical  Support 

Number  Sold:  25,000+  Single  User  Price:  $795  Phone:  503-362-9131  Fax:  — 

Report  Date:  1/07/93  Report  Updated:  5/16/94  E-mail: 

Training  Available  Address:  245  Court  St.  NE 

Evaluation  Copy  Available  Site  License  Available  Salem,  OR  97301 

Description:  FEU  is  an  AS/400  or  S/36  software  utility  that  allows  users  to  access  any  file  without  any  setup  or  programming.  Allows 
additions,  deletions,  and  changes  under  various  "formats."  Includes  global  search/replace,  audit  report  and  custom  capabilities. 
Classification:  Coding,  Testing,  Quality  Assurance,  Reengineering,  Database,  Auditor,  Data  Extractor,  Debugger,  Test  Data  Generator 
Features: 

Languages  Supported:  Assembler,  RPG  n 
Configurations:  AS/400,  SYSTEM  36 


Tool:  FIRSTCASE 

Version:  3.0 
Number  Sold:  55 
Report  Date:  1/07/93 
Training  Available 


Vendor: 


Release  Date:  10/01/92 
Single  User  Price:  $15K-$70K 
Report  Updated:  7/0 1  /94 


POC 

Phone 

E-mail 

Address 


AGS  Management  Systems 
Mike  Marcus 

800-678-8484  Fax: 


215-265-1230 


Training  Available  Address:  880  First  Ave. 

King  of  Prussia,  PA  19406 

Description:  FIRSTCASE  is  a  fully  automated  PC-based  support  framework  for  systems  development  that  includes  process 
management  (Systems  Development  Methodology),  estimating  support,  project  management,  CASE  tool  development  management, 
and  deliverables  management.  It  provides  tools,  techniques,  and  procedures  that  are  used  over  the  system  development  lifecycle,  and  its 
methodology  guides  the  development  team  in  the  proper  selection,  use,  and  performance  of  those  tools,  techniques,  and  procedures. 

The  estimation  capabilities  include  function  point  analysis.  FIRSTCASE  is  designed  to  complement  existing  CASE  tools.  It  provides  a 
shell  or  platform  into  which  the  user  can  plug  in  CASE  tools  of  the  user's  choice,  thereby  supporting  the  "best  of  breed"  selection  of 
CASE  tools.  FIRSTCASE  incorporates  five  standard  system  lifecycles,  including  one  for  maintenance  and  another  for  purchased 
software. 

Classification:  Coding,  Project  Management,  Configuration  Management,  Quality  Assurance,  Metrics,  Reengineering,  Data  Reducer 
&  Analyzer,  Size  Measurer 

Features:  Function  Points 

Languages  Supported:  - 

Configurations:  OS/2,  Windows,  PC/MS-DOS  _  _ 
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Tool:  gprof 

Vendor: 

Digital  Equipment  Corp. 

Version:  --  Release  Date: 

POC: 

Technical  Support 

Number  Sold:  -  Single  User  Price:  (Contact  Vendor) 

Phone: 

800-344-4825  Fax:  603-881-2381 

Report  Date:  1/07/93  Report  Updated:  7/01/94 

E-mail: 

Description:  gprof  provides  call  profile/count  information. 
Classification:  Coding,  Reengineering,  Structure  Checker 

Features: 

Languages  Supported:  C 

Configurations:  DECstation/ULTRlX 

Address: 

146  Mairr  St. 

Maynard,  MA  01754-2571 

Tool:  GPSA 

Version:  2.0 
Number  Sold:  4  Sites 
Report  Date:  1/27/93 


Release  Date:  11/01/91 
Single  User  Price:  $8,000 
Report  Updated:  7/01/94 


Vendor: 

POC: 

Phone: 

E-mail: 

Address: 


COBOL  Maintenance  Technologies 
Gordon  Pannell 

619-429-0121  Fax:  - 

PO  Box  122069 
Chula  Vista,  CA  91912 


Evaluation  Copy  Available  Site  License  Available  Chula  Vista,  CA  91912 

Description:  GPSA  is  a  system-wide  COBOL  analysis  and  documentation  toolset.  An  interactive  viewing  facility  and  26  reports 
provide  complex  cross-reference  of  every  program,  copybook,  paragraph  and  data-name. 

Classiflcation:  Coding,  Documentation,  Quality  Assurance,  Reengineering,  Reuse,  Cross  Referencing  Tool,  Reusable  Components 
Identifier,  Reverse  Engineering,  Syntax  &  Semantics  Analyzer 

Features: 

Languages  Supported:  COBOL 
Configurations:  MS-DOS  3.3 


Release  Date:  2/01/92 
Single  User  Price:  $5,000 
Report  Updated:  7/01/94 


Software  Systems  Design,  Inc. 
Dr.  Thomas  S.  Radi 
714-625-6147  Fax: 

3627  Padua  Ave. 

Claremont,  CA  91711 


714-626-9667 


Tool:  GrafBrowse  V Clldor  I  Software  Systems  Design,  Inc. 

Version:  2.3  Release  Date:  2/01/92  POC:  Dr.  Thomas  S.  Radi 

Number  Sold:  20  Single  User  Price:  $5,000  Phone:  714-625-6147  Fax:  714-626-9667 

Report  Date:  6/28/93  Report  Updated:  7/01/94  E-mail: 

Address:  3627  Padua  Ave. 

Evaluation  Copy  Available  Site  License  Available  Claremont,  CA  91711 

Description:  GrafBrowse  is  a  graphical  reverse  engineering  system  and  interactive  browser  for  Ada,  C,  and  FORTRAN  source  codes 
and  design.  It  analyzes  codes  for  certain  metrics. 

Classification:  Design,  Coding,  Documentation,  Quality  Assurance,  Metrics,  Reengineering,  Complexity  Measurer,  Cross  Referencing 
Tool,  Reverse  Engineering,  Size  Measurer 
Features:  SLOC  Actuals 
Languages  Supported:  Ada,  c,  FORTRAN 

Configurations:  Apollo,  VMS,  DECstation  UNIX,  Gould,  Harris  SPARC,  HP,  MIPS,  PC  MS-DOS,RISC  6000  _ 


Tool:  HARMONIZER 

Vendor: 

Aldon  Computer  Group 

Version:  - 

Release  Date: 

POC: 

Technical  Support 

Number  Sold:  - 

Single  User  Price: 

(Contact  Vendor)  Phone: 

510-839-3535  Fax:  510-839-2894 

Report  Date:  1/07/93 

Report  Updated: 

E-mail: 

Address: 

401  15th  St. 

Oakland,  CA  94612 

Description:  HARMONIZER  compares  files  to  determine  differences. 

Classification:  Testing,  Comparator 

Features: 

Languages  Supported: 
Configurations: 

All 
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Tooh  InnerVue  Vendor  J  Performance  Awareness  Crop. 

Version:  --  Release  Date:  POC:  Technical  Support 

Number  Sold:  -  Single  User  Price:  (Contact  Vendor)  Phone:  801-974-1700  Fax:  801-974-1780 

Report  Date:  1/07/93  Report  Updated:  6/01/94  E-mail: 

Address:  3530  W.  2100  S. 

Salt  Lake  City,  UT  84119 

Description:  InnerVue  provides  a  unique  real-time  view  of  many  of  the  different  aspects  of  a  UNIX  systems  performance.  Over  30 
different  metrics  may  be  monitored  and  displayed  in  a  graphical  format  on  a  PC. 

Classification:  Testing,  Performance/Timing  Analyzer 

Features: 

Languages  Supported:  - 

Configurations:  PC/MS-DOS _ _ _ 

Tool!  Insight  Vcildor!  Rational 

Version:  —  Release  Date:  POC:  Dan  Maes 

Number  Sold:  -  Single  User  Price:  (Contact  Vendor)  Phone:  303-986-2006  Fax:  303-986-0205 

Report  Date:  1/07/93  Report  Updated:  E-mail: 

Address:  165  So.  Union  Blvd.,  Suite  604 
Lakewood,  CO  80228 

Description:  Insight  allows  you  to  reengineer  and  maintain  Ada  software.  Insight  gathers  information  about  the  code  and  displays 
both  object-oriented  module  diagrams. 

Classification:  Coding,  Documentation,  Reengineering,  Reverse  Engineering,  Structure  Checker 

Features: 

Languages  Supported:  Ada 

Configurations:  RISC  6000,  Sun _ 

Tool:  Instant'C  Vendor:  Rational  Systems,  Inc. 

Version:  5.0  Release  Date:  1/01/92  POC:  Technical  Support 

Number  Sold:  -  Single  User  Price:  (Contact  Vendor)  Phone:  508-653-6006  Fax:  508-655-2753 

Report  Date:  1/07/93  Report  Updated:  5/23/94  E-mail: 

Address:  220  N.  Main  St. 

Natick,  MA  01760 

Description:  Instant-C  allows  the  user  to  generate  high-quality  code  quicker  and  more  efficiently.  It  increases  productivity  by 
combining  a  linker  and  an  incremental  compiler  with  automatic  static  and  run-time  error  detection. 

Classification:  Coding,  Testing,  Compiler,  Cross  Referencing  Tool,  Debugger,  Linker,  Recompiler,  Simulator,  Syntax  &  Semantics 
Analyzer 

Features:  Run-Time  Error  Detection 

Languages  Supported:  c 

Configurations:  PC _ _ _ 


Tool!  IntegrAda 

Vendor:  Aetech 

Version:  - 

Release  Date: 

POC:  Technical  Support 

Number  Sold:  - 

Single  User  Price:  $95 -$695 

Phone:  303-986-2006  Fax:  303-986-0205 

Report  Date:  3/03/93 

Report  Updated:  5/16/94 

E-mail: 

Training  Available 

Newsletter  Available 

Address:  165  So.  Union  Blvd.,  Suite  604 

Site  License  Available 

Lakewood,  CO  80228 

Description:  IntegrAda  is 

an  integrated  Ada  Programming  Support  Environment  (APSE)  with  a  production-quality  compiler  that  has 

been  fully  validated  by  the  gov't  to  the  latest  Ada  Compiler  Validation  Capability  (ACVCl.lO). 

Classification:  Coding,  Documentation,  Reengineering,  Software  Engineering  Environment,  Code  Generator,  Commenter,  Compiler, 

Forward  Engineering,  Linker,  Reformatter,  Session  Documenter,  Structure  Checker 

Features:  Ada  language  sensitive  editing 

Languages  Supported: 

Ada 

1  Configurations:  PC/MS-DOS 
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Tool:  Micro  Focus  Toolbox  for  UNIX  Vendor: 

Version:  --  Release  Date:  POC: 

Number  Sold:  -  Single  User  Price:  (Contact  Vendor)  Phone: 

Report  Date:  12/28/92  Report  Updated:  6/02/94  E-mail: 

Address: 


Micro  Focus,  Inc. 
Guy  J.  Spaniak 
415-856-4161 


415-856-6134 


Address:  2465  E.  Bayshore  Rd.,  Suite  400 
Palo  Alto,  CA  94303 

Description:  Toolbox  for  UNIX  is  a  powerful  set  of  utilities  designed  to  bring  increased  productivity  to  the  UNIX/COBOL 
development  environment. 

Classiflcation:  Design,  Coding,  Testing,  Quality  Assurance,  Reengineering,  Capture-Replay  Tool,  Coverage/Frequency  Analyzer, 
Debugger,  Language  Sensitive  Editor,  Performance/Timing  Analyzer,  Screen  Painter,  Structure  Checker 

Features: 

Languages  Supported:  COBOL  n 
Configurations:  UNIX 


Tool:  Micro  Focus  Workbench  Vendor:  Micro  Focus,  Inc. 

Version:  3.0  Release  Date:  POC:  Technical  Support 

Number  Sold:  ~  Single  User  Price:  (Contact  Vendor)  Phone:  415-856-4161  Fax:  415-856-6134 

Report  Date:  11/22/91  Report  Updated:  7/05/94  E-mail: 

Address:  2465  E.  Bayshore  Rd.,  Suite  400 
Palo  Alto,  CA  94303 

Description:  Micro  Focus  Workbench  provides  advanced  visual  productivity  tools,  program  debugging  and  construction,  program 
analysis,  and  navigation.  It  also  has  a  file  maintenance,  conversion,  and  transfer. 

Classification:  System  Simulation,  Design,  Coding,  Testing,  Quality  Assurance,  Reengineering,  Database,  Software  Engineering 
Environment,  Capture-Replay  Tool,  Coverage/Frequency  Analyzer,  Data  Extractor,  Debugger,  Language  Sensitive  Editor, 
Performance/Timing  Analyzer,  Screen  Painter,  Test  Execution  Manager 

Features: 

Languages  Supported:  COBOL  n 

Configurations:  PC  _  _ 


Tool:  Microsoft  Source  Profiler 

Version:  —  Release  Date: 

Number  Sold:  -  Single  User  Price: 

Report  Date:  8/05/93  Report  Updated: 


•rofiier  Vcodor: 

Release  Date:  POC 

Single  User  Price:  (Contact  Vendor)  Phone 

Report  Updated:  7/05/94  E-mail 

Address 


Microsoft  Corp. 
Tech.  Support  (MiCo) 
800-426-9400 


Fax:  617-740-0025 


Address:  1  Microsoft  Way 

Redmond,  WA  98052-6399 

Description:  Microsoft  Source  Profiler  generates  statistical  profile  information  for  DOS,  OS/2,  and  Windows-based  programs. 
Classification:  Testing,  Performance/Timing  Analyzer 

Features: 

Languages  Supported:  - 
Configurations:  OS/2,  PC/MS-DQS 


Toolj  Microsoft  Test 

Vendor: 

Microsoft  Corp. 

Version:  - 

Release  Date: 

POC: 

Tech.  Support  (MiCo) 

Number  Sold:  — 

Single  User  Price:  (Contact  Vendor) 

Phone: 

800-426-9400  Fax;  617-740-0025 

Report  Date:  3/01/93 

Report  Updated:  7/05/94 

E-mail: 

Address: 

1  Microsoft  Way 

Redmond,  WA  98052-6399 

Description:  Microsoft  Test  is  an  automated  testing  tool  used  to  validate  the  quality  of  Microsoft  Window  applications.  It  can  test  any 

Windows  application. 

Classification:  Testing,  Capture-Replay  Tool,  Comparator 

Features: 

Languages  Supported: 

1  Configurations:  PC/Window  3.0 
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Tool:  MicroSTEP  Vendor: 

Version:  1.6  Release  Date:  3/01/92  POC: 

Number  Sold:  ~  Single  User  Price:  $895  Phone: 

Report  Date:  1/26/93  Report  Updated:  7/05/94  E-mail: 

Address: 

Description:  MicroSTEP  is  a  "visual  programming"  application.  A  supports  d 
Classiflcation:  Design,  Requirements  Analysis,  Reengineering,  DefecVChang 
Checker 

Features: 

Languages  Supported:  c 

Conflgurations:  PC/MS-DOS 

SYSCORP  Inti. 

Holly  Nash 

800-727-7837  Fax:  512-338-5810 

9430  Research  Blvd.,  Echelon  IV,Suite  300 

Austin,  TX  78759 

esign  validation  and  quality  domain  documentation, 
e  Tracker,  Forward  Engineering,  Run-Time  Error 

Tooll  Microtec  Cross  Development  Tools  Vcildor; 

Version:  4.2  Release  Date:  POC: 

Number  Sold:  -  Single  User  Price:  (Contact  Vendor)  Phone: 

Report  Date:  12/28/92  Report  Updated:  6/02/94  E-mail; 

Training  Available  Newsletter  Available  Address: 

Evaluation  Copy  Available 

Description:  The  Microtec  Research  Cross  Development  Tools  form  an  advar 
numerous  features  that  are  required  for  developing  high-performance  embeddec 
Classiflcation:  Coding,  Testing,  Quality  Assurance,  Reengineering,  Assemble 
Assembler,  Debugger,  Emulator,  Linker,  Performance/Timing  Analyzer,  Simul 

Features: 

Languages  Supported:  Assembler,  C,  C++ 

Conflgurations:  PC/MS-DOS,  Sun/SPARC 

Microtec  Research,  Inc. 

Jeff  Shimbo 

800-950-5554  Fax:  408-982-8266 

2350  Mission  College  Blvd. 

Santa  Clara,  CA  95054 

iced  software  development  toolkit.  The  toolkit  provides 
i  applications. 

ir.  Compiler,  Coverage/Frequency  Analyzer,  Cross 
ator,  Structure  Checker 

Tool;  MICS  MVS  Performance  Manager  Vcndori 

Version:  —  Release  Date;  POC 

Number  Sold:  ~  Single  User  Price:  (Contact  Vendor)  Phone 

Report  Date:  12/29/92  Report  Updated:  7/05/94  E-mail 

Address 

Description:  MICS  MVS  Performance  Manager  furnishes  facilities  for  MVS  ! 

configuration  analysis  functions.  It  classifies  workloads  and  identifies  bottlene< 
Classiflcation:  Testing,  Quality  Assurance,  X  OS  Utility,  Performance/Timir 

Features: 

Languages  Supported:  - 
Configurations:  MVS 

Legent  Corp. 

Steven  King 

800-726-1637  Fax:  508-836-5992 

575  Herndon  Parkway 

Herndon,  VA  22070 

system  tuning,  batch  workload  analysis,  and  I/O 
cks  as  well  as  providing  reporting  capabilities, 
ig  Analyzer 

- - - — — - — . . . — - f 

Tool:  Monitor/Plus  Vcildorj  Data  Center  Software 


Version:  2.0  Release  Date:  6/15/92  POC:  Dave  Powers 

Number  Sold:  -  Single  User  Price:  (Contact  Vendor)  Phone:  800-726-1637  Fax:  508-836-5992 

Report  Date:  12/23/92  Report  Updated:  7/05/94  E-mail: 

Address:  575  Herndon  Parkway 
Herndon,  VA  22070 

Description:  MONITOR/Plus  helps  to  track  key  usage  and  performance  statistics,  taking  specified  action  on  your  behalf  when  a 
problem  condition  arises.  Allows  customized  messages  and  automated  personnel  notification. 

ClaSSiHcation:  Testing,  Performance/Timing  Analyzer 

Features: 

Languages  Supported:  - 

Configurations:  VAX/VMS _ 
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Tool;  PROBLEM  PROGRAM  EVALUATION  (PPE)  Vcndor: 

Version:  -  Release  Date:  POC: 

Number  Sold:  --  Single  User  Price:  (Contact  Vendor)  Phone: 

Report  Date:  3/15/93  Report  Updated:  5/12/94  E-mail: 

Address: 


Boole  &  Babbage,  Inc. 
Technical  Support 
415-788-4700 


415-788-1144 


Address:  601  Montgomery  St,,  Suite  675 
San  Francisco,  CA  94111 

Description:  PROBLEM  program  EVALUATION  (PPE)  performs  system  analysis,  workload  analysis,  and  program  analysis, 
provi^ng  data  on  excessive  CPU  usage  for  both  system  modules  and  application  program  modules. 

Classiflcation:  System  Simulation,  Testing,  Quality  Assurance,  Performance/Timing  Analyzer 

Features: 

Languages  Supported:  COBOL,  COBOL  n 
Configurations:  MVS,XA 


Toolj  PRODOC  re/NuSys  Workbench 

Version:  Release  Date: 

Number  Sold:  -  Single  User  Price: 

Report  Date:  4/07/93  Report  Updated: 
Training  Available  Newsletter 


s  Workbench  Vendor: 

Release  Date:  1/01/93  POC: 

Single  User  Price:  (Contact  Vendor)  Phone: 

Report  Updated:  5/16/94  E-mail: 

Newsletter  Available  Address: 


SCANDURA  &  Assoc. 
Alice  B.  Scandura 
215-664-1207 


215-664-7276 


Training  Available  Newsletter  Available  Address:  822  Montgomery  Ave.  Suite  317 

Narberth,  PA  19072 

Description:  PRODOC  re/NuSys  Workbench  is  a  fully  integrated,  full  lifecycle  design,  development  and  maintenance  system  for 
several  Languages.  PRODOC  is  also  a  translator  (C-Ada,  Cobol-Ada,  Cobol-C,  Fortran-Ada,  Fortran-C,  Pascal-Ada,  and  Pascal-C). 
Classification:  System  Simulation,  Requirements  Trace,  Design,  Coding,  Requirements  Analysis,  Documentation,  Quality  Assurance, 
Metrics,  Reengineering,  Reuse,  Database,  Software  Engineering  Environment,  Complexity  Measurer,  Coverage/Frequency  Analyzer, 
Cross  Referencing  Tool,  Forward  Engineering,  Redocumenter,  Restructurer,  Reusable  Components  Identifier,  Reverse  Engineering, 
Simulator,  Source  Code  Translator,  Structure  Checker,  Syntax  &  Semantics  Analyzer 

Features: 

Languages  Supported:  Ada,  C,  C++,  COBOL,  FORTRAN,  Pascal,  PL/1 

Configurations:  UNIX,  DEC/OSF/l,  IBM/AIX,  PC/MS-DOS,  PC/OS/2,  Sun/Solaris2.1,  Sun/SPARC,  Sun/Sun  OS 


T OOl :  Productivity  Manager  V eildor :  Productivity  Management  Group,  Inc. 

Version:  2.4  Release  Date:  1/01/93  POC:  Walter  E.  Lopus 

Number  Sold:  -  Single  User  Price:  $10,000  Phone:  716-689-7724  Fax:  - 

Report  Date:  2/26/93  Report  Updated:  7/06/94  E-mail: 

Training  Available  Address:  178  Foxhunt  Lane 

Evaluation  Copy  Available  Site  License  Available  East  Amherst,  NY  1405 1 

Description:  Productivity  Manager  provides  organizations  with  a  flexible  and  comprehensive  structure  to  evaluate  their  software 
engineering  processes.  It  stores  detailed  and  summary  function  point  counts  as  well  as  costs  defects,  time,  environment,  tools  and 
techniques. 

Classification:  Requirements  Analysis,  Project  Management,  Quality  Assurance,  Metrics,  Complexity  Measurer,  Size  Measurer 
Features:  Function  Points 
Languages  Supported:  -- 
Configurations:  PC  _ 


Tool:  prof 

Vendor: 

Digital  Equipment  Corp. 

Version:  -  Release  Date: 

POC: 

Technical  Support 

Number  Sold:  -  Single  User  Price:  (Contact  Vendor) 

Phone: 

800-344-4825  Fax:  603-881-2381 

Report  Date:  3/1 5/93  Report  Updated: 

E-mail: 

Description:  This  is  a  syntax  and  semantics  analyzer. 

Address: 

146  Mairr  St. 

Maynard,  MA  01754-2571 

Classification:  Coding,  Reengineering,  Syntax  &  Semantics  Analyzer 

Features: 

Languages  Supported:  c 

Configurations:  DECstation/ULTRIX 

AA25 


Software  Technology  Support  Center 


A-126 


Appendix  A.l :  Product  Sheets  by  Tool  Name 


A-127 


Software  Technology  Support  Center 


A-128 


Appendix  A.1:  Product  Sheets  by  Tool  Name 


A-129 


Software  Technology  Support  Center 


A-130 


Appendix  A.1:  Product  Sheets  by  Tool  Name 


A-131 


Software  Technology  Support  Center 


A-132 


Appendix  A.l:  Product  Sheets  by  Tool  Name 


A-133 


Software  Technology  Support  Center 


A-134 


Appendix  A.l:  Product  Sheets  by  Tool  Name 


A-135 


Software  Technology  Support  Center 


A-137 


Software  Technology  Support  Center 


Appendix  A.l:  Product  Sheets  by  Tool  Name 


A-139 


Software  Technology  Support  Center 


A-140 


Appendix  A.l:  Product  Sheets  by  Tool  Name 


Software  Technology  Support  Center 


A-142 


Appendix  A.l:  Product  Sheets  by  Tool  Name 


Software  Technology  Support  Center 


A-144 


Appendix  A. 1:  Product  Sheets  by  Tool  Name 


A-145 


Software  Technology  Support  Center 


Appendix  A.  J:  Product  Sheets  by  Tool  Name 


A-147 


Software  Technology  Support  Center 


A-148 


Appendix  A.1:  Product  Sheets  by  Tool  Name 


A-149 


Software  Technology  Support  Center 


A-150 


Appendix  A.l:  Product  Sheets  by  Tool  Name 


Software  Technology  Support  Center 


Appendix  A.l:  Product  Sheets  by  Tool  Name 


A-153 


Software  Technology  Support  Center 


A-154 


Appendix  A.l :  Product  Sheets  by  Tool  Name 


Software  Technology  Support  Center 


A- 156 


Appendix  AJ:  Product  Sheets  by  Tool  Name 


Software  Technology  Support  Center 


A- 158 


Appendix  A. 1 :  Product  Sheets  by  Tool  Name 


A-159 


Software  Technology  Support  Center 


A-160 


Appendix  A.l :  Product  Sheets  by  Tool  Name 


Tool:  xREFPLus  Vendor:  Jensen  Research  Co. 

Version:  -  Release  Date:  POC:  Charles  J.  Smith 

Number  Sold:  -  Single  User  Price:  (Contact  Vendor)  Phone:  201-337-4000  Fax:  201-337-4334 

Report  Date:  3/02/93  Report  Updated:  7/06/94  E-mail: 

Address:  169  Ramapo  Valley  Road 
Oakland,  NJ  07436 

Description:  XREFPLUS  famishes  a  large  variety  of  highly  detailed  reports  that  include  DATASETs,  MEMBER  FLOW  CHARTS, 
NOT-FOUND  SUMMARYs,  and  more.  Makes  customized  reports. 

Classification:  Coding,  Testing,  Documentation,  Reengineering,  Cross  Referencing  Tool 

Features: 

Languages  Supported:  JCL 

Configurations:  IBM/MVS _ 

Tool  I  XRunner  Voildor;  Mercury  Interactive  Corp. 

Version:  2.1  Release  Date:  1/01/92  POC:  Stephanie  Froude 

Number  Sold:  9000  Single  User  Price:  (Contact  Vendor)  Phone:  408-982-0100  Fax:  408-982-0149 

Report  Date:  12/28/92  Report  Updated:  7/06/94  E-mail:  steph@merc-mt,com 

Training  Available  Address:  3333  Octavius  Dr.  Suite  104 

Evaluation  Copy  Available  Site  License  Available  Santa  Clara,  CA  95054 

Description:  XRunner  is  a  testing  system  for  UNDC/X  Windows  environment  and  is  suited  for  testing  stand-alone  applications  or 
client/server  applications  in  a  single-client  configuration.  XRunner  addresses  the  growing  need  for  improvements  in  test/QA 
productivity. 

Classification:  Coding,  Testing,  Quality  Assurance,  Reuse,  Capture-Replay  Tool,  Data  Reducer  &  Analyzer,  Maintainability 
Analyzer,  Reliability  Analyzer,  Test  Data  Generator,  Test  Execution  Manager 

Features:  GUI-based  Testing 

Languages  Supported:  All 

Configurations:  DEC/ULTRix,  hp/hp-ux,  ibm,  Sun/Sun  os _ _ 
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+1  Software  Engineering 

A+  Software,  Inc. 

ABRAXAS  Software,  Inc. 

AccelrS  Technology  Corp. 
AccuWare,  Inc. 

Advanced  Programming  Techniques 
Advanced  Software  Automation,  Inc. 


Advanced  Software,  Inc. 

Advanced  Systems  Concepts,  Inc. 

Aetech 

AGS  Management  Systems 
Aldon  Computer  Group 
Allen  Systems  Group 

ALSYS,  Inc. 

Amadeus  Software  Research,  Inc. 
American  InterFace  Computer,  Inc. 
Analysis  &  Computer  Systems,  ftic. 
Applied  Business  Technology 
Applied  Logic  Corp. 

Archimedes  Software,  Inc. 

Array  Systems  Computing  Products,  Inc. 
ASTA,  Inc. 

ASV/SCEL 

AT&T,  Performance  Analysis  and  Tools 
AT&T,  Software  Solutions  Group 
AutoCASE  Technology 
AutoTester,  Inc. 

B-TREE  SOFTWARE,  Inc. 

BBN  Systems  and  Technologies 
Bell  Canada,  Corp.  Quality  Assurance 
Bender  &  Assoc.,  Inc. 

BGS  Systems 
Biomation 

Blossom/Catalytix  Corp. 

Boole  &  Babbage,  Inc. 

Bullseye  Software 
Cadre  Technologies,  Inc. 

CARDtools  Systems 
Cater  Software 
CenterLine  Software,  Inc. 

CGI  Systems,  Inc. 

Cincom  Systems,  Inc. 

Clarity  Concepts  Systems 
CLEAR  Software,  Inc. 


Metrics4C,  Metrics4Fortran,  Metrics4Pascal,  Metrics4Project, 
TREESOFT 

Automated  Documentation  System  (ADS),  Navigator 

CodeCheck,  PCYACC 

OpenACCLIMS 

AccuTest 

SEEK 

AutoAnalyzer,  AutoDiagrammer,  AutoStructureChart,  C  Source 
Analyzer,  Hindsight,  Software  Quality  Analysis  Module,  Structure 
&  Logic  Analysis  Module,  Test  Coverage  Analysis  Module 
DocuComp 

ABSTRACT/PROBE,  STATUS 

Ada  Software  Development  Toolset,  IntegrAda,  XAda 

FIRSTCASE 

Analyzer,  COMPARE,  HARMONIZER,  SCOMPARE 
AdA^antage.DM*,  Documentation-Aid,  MVS  (Doc-Aid), 
SEDIT.DB,  SEDIT.MVS,  TestBench.DM* 

AdaProbe/ICE,  AdaTune,  AdaVerify 

Amadeus 

PLASMA 

CaseQMS,  CustomQA 
Metrics  Manager  (M  M) 

File  Edit  Utility  (FEU) 

BugBase 

Expert  Debugging  Software  Assistant  (EDSA) 

QA  Administer,  QA  C,  QA  C++,  QA  FORTRAN,  QA  Process 
Control  Manager  (QA  PCM) 

J73  Automated  Verification  System  (J73AVS) 

BUSTER  Test  Management  System 
QUARTZ  Remote  Terminal  Emulator 
AutoFlow 

AutoTester,  AutoTester  For  OS/2,  AutoTester  for  Windows, 
AutoTester  Plus 

esVS,  SVS  Automated  Software  Testing 

BBN/Catalyst  Software 

DATRIX 

SOFTTEST 

CRYSTAL 

CLAS  2000,  Variable  Value  Monitor  (VVM-1) 

Safe  C  Runtime  Analyzer 

PROBLEM  PROGRAM  EVALUATION  (PPE) 

C-Cover 

Ensemble,  Teamwork/Ada,  Teamwork/TestCase 
CARDtools  (CARDtools) 

C->it  (see.it),  C/ANALYST 
CodeCenter,  ObjectCenter,  TestCenter 
PACREVERSE,  Source/RE 
XpertRule 

Enforcer  I,  Enforcer  II 
CLEAR  Plus 
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Tool  Name 


Clyde  Digital  Systems,  Inc. 

Cobalt  Blue,  Inc. 

COBOL  Maintenance  Technologies 
Cole  Software,  Inc. 

Computer  Associates  Inti. 


Computer  Data  Systems,  Inc. 
Computer  Innovations,  Inc. 
Computer  Power  Group,  Inc. 
Computer  Sciences  Corp. 
Compuware  Corp. 

Concurrent  Computer  Corp. 
Consumer  Systems  Corp. 
Corporate  Computer  Systems 
COSMIC 

Dashboard  Software 
Data  Center  Software 
Databorough,  Inc. 

DataMetrics  Systems  Corporation 
DiagSoft,  Inc. 

Digital  Equipment  Corp. 


Digital  Sciences,  Inc. 

Direct  Technology 

Donatech  Corp. 

Dynamics  Research  Corp. 

Eastern  Systems,  Inc. 

Eden  Systems  Corp. 

EDP  Management,  Inc. 

EEsof,  Inc. 

EVB  Software  Engineering,  Inc. 

Franz  Inc. 

Galorath  Associates,  Inc. 

Gary  Bergman  Associates,  Inc. 

General  Electric  Co.  (R  &  D) 

General  Research  Corp. 

Gimpei  Software 

GPP  Ges  fur  Prozessrechnerprogrammierung 
Hertzler  Systems,  Inc. 

Hewlett-Packard 


CarbonCopy 

FOR.STRUCT,  FOR.STUDY 

GPSA 

CPR,  XDC 

CA-COBOLVISION/Analyzer,  CA-EZTEST/CICS,  CA-FPXpert, 
CA-InterTest,  CA-METRICS,  CA-OPTIMIZER,  CA-PAN/LCM, 
CA-Realia  II  Workbench,  CA-TRAPS,  CA-VERIFY 
COBOL/METRICS,  CONFIGURE,  SCAN/COBOL,  SLEUTH 
MAX  Macro  Assembler  (MAX) 

Metrics  Analysis  and  Reporting  System  (MARS),  Test/Cycle 
DESIGN  GENERATOR 

Compuware  Solution  Set,  DATATEC,  PATHVU,  PLAYBACK 
ADANON,  ADAXPA,  ADAXREF,  C3 Ada  Symbolic  Debugger 
DataBasic  n 
DELTA,  SCONS 
FPT 

TrackDeck 

Monitor/Plus,  QUEMAN 

X-ANALYSIS 

Viewpoint 

QAPlus 

cflow,  COHESION  Team/SEE,  ctrace,  cxref,  DEC  FUSE,  DEC 
FUSE  Call  Graph  Browser,  DEC  FUSE  Cross-Reference, 

DEC/Test  Manager  (DTM),  DECset,  diff  in  Op  Sys,  dxdiff,  gprof, 
lint,  perfmeter,  perfmon.  Performance  Coverage  Analyzer  (PCA), 
pixie,  prof,  size,  time,  VAX  Language  Sensitive  Editor  (LSE), 

VAX  Performance  Advisor  (VP A),  VAX  Source  Code  Analyzer 
(SCA),  VAXset/Program  Design  Facility  (VAXset/PDF),  vtimes 
Fortran  Utility  System 

Automator  QA,  Automator  Q A  (Windows  Edition),  Automator 
QA  with  Navigator 
RealTime  Testware  (RT) 

Ada  Measurement  and  Analysis  Tool  (AdaMAT) 

Evaluator 

ES  REmsion,  Q/Artisan,  Q/ARTISAN  PC,  Q/AUDITOR  PL/I, 

Q/AUDITORPL/IPC 

LOOKAT 

Anacat 

Complexity  Measures  Tool  (CMT) 

Allegro  Composer  Development  Environment 
SEER 

Advanced  Debugging  System  (ADS) 

Environment  for  Code  Re-Engineering  (ENCORE) 

AdaQuest,  RXVP80 

C- Vision  for  C,  FlexeLint,  PC-Lint 

RE-DOC,  RE-SPEC 

QA/S 

Basis  Branch  Analyzer  (BBA),  HP  640(X)-UX  Micro.  Software 
Dev.,  HP  Ada/300  Development  System,  HP  AxAda  Programming 
Support  Environment,  HP  Branch  Validator,  HP  Softbench 


A-165 


Software  Technology  Support  Center 


Vendor  Name 


Tool  Name 


Hypersoft  Corp. 

Mormation  Processing  Techniques  Corp. 
Ihtek  Integration  Technologies 
Intel  Corp. 

Interactive  Development  Environments 
Interlex 

International  Software  Automation,  Inc. 


ftiterPort  Software  Corp. 
Intersolv 

Inti.  Business  Machines  Corp. 


Inti.  Logic  Corp. 

Inti.  Software  Systems,  Inc. 

IPL  Information  Processing  Ltd. 

ITCN 

Jacobson  Software,  Inc. 

Jensen  Research  Co. 

Keithley  Instruments,  Inc. 

Knowledge  Ware,  ftic. 

Legent  Corp. 

LIANT  Software  Corp. 

Little  Tree  Consulting 
LOGICON,  Inc. 

M.D.  Friedman  Associates,  Inc. 
MacKinney  Systems 
Marble  Computer,  Inc. 

Marconi  Systems  Technology 
MB  Solutions,  Inc. 

mbp  Software  &  Systems  Technology,  Inc. 
McCabe  &  Assoc.,  Inc. 

Menlo  Business  Systems,  Inc. 

Mercury  Interactive  Corp. 

Micro  Focus,  Inc. 


Microsoft  Corp. 

Microtec  Research,  Inc, 
MultiScope 

National  Information  Systems,  Inc. 


NCCOSC 


Neal  Nelson  &  Assoc. 
NOI  Systems 


Application  Browser 

COBOL-lint,  FORTRAN-lint,  lint-PLUS 

ftitek  C*H- 

SimCASE  -  Simulator/Debugger  (SimCASE) 

Software  through  Pictures  (StP),  StP/T,  T 
Source  Code  Analyzer  (SCA) 

Panorama  C,  Panorama  C++,  Panorama  C++/00- Analyzer, 
Panorama  C++/00-Browser,  Panorama  C++/00-Diagrammer, 
Panorama  C++/00-SQA,  Panorama  C++/00~Test,  QualityFirst  C 
InterCASE  Knowledge  Ware  Gateway  (IKG) 

Design  Recovery  Series,  Excelerator  for  Design  Recovery 
Automated  Software  Test  Facility,  Cross  System  Product  (CSP), 
Software  Analysis  Test  Tool,  VS  FORTRAN 
ComplierA-ibrary/Debug,  Workstation  Interactive  Test  Tool 
(WITT) 

F-SCAN 
Auto  V&V/Ada 
AdaTEST,  CANTATA 

Computer  Tester  Analyzer  Controller  (C-TAC),  TD20(X) 

COBOL  Magic 

XREFPLUS 

TestPoint 

ADW/Inspector,  ADW/Pinpoint,  Legacy  Workbench 
ENDEVOR,  MICS  MVS  Performance  Manager,  Problem  Alert 
System  (PAS) 

C-SCAPE 

Ada  Analyzer,  Ada  Type  Interchange  Generator 
Static  Analysis  Tool  Set  (SATS) 

COBOL  STANDARDS  ANALYZER,  CONVERSION  ENGINE 
COBOL  Glossary,  Source  Program  Compare 
DCD  III 

Requirements  and  Traceability  Management  (RTM) 

JCL/Convert,  JCL/Cross-Reference 
Visual  COBOL 

Battlemap  Analysis  Tool  (BAT),  CodeBreaker,  McCabe 
Instrumentation  Tool,  McCabe  Slice  Tool,  Slice  Tool 
Foundation  VistalO 

LoadRunner,  TestRunner,  WinRunner,  XRunner 
Micro  Focus  COBOL/2  for  UNIX,  Micro  Focus  Cobol/2 
Workbench,  Micro  Focus  Toolbox  for  UNIX,  Micro  Focus 
Workbench 

Microsoft  Source  Profiler,  Microsoft  Test 

Microtec  Cross  Development  Tools,  XRAY  Source  Explorer 

Run  Time  Debugger 

ACCENT  R 

CMS-2  Design  Analyzer  (DESAN),  CMS-2  Standards  Checker 
(STDCK),  CMS-2  Test  Coverage  Analyzer  (TCA),  Metrics 
Generator  (METRC) 

Business  BenchMark,  Remote  Terminal  Emulator  (RTE) 
SHOWDIFF 
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Norwegian  Technical  University 
Nu-Mega  Technologies 
Oasys,  Inc. 

Onset  Computer  Corp. 

Optima  Software,  Inc. 

C^timization  Technology,  Inc. 

OTG  Systems,  Inc. 

ParcPlace  Systems 
Peregrine  Systems,  Inc. 

Perennial  Corp. 

Performance  Awareness  Crop. 
Performance  Software,  Inc. 

Personyx 
ProCASE  Corp. 

Productivity  Management  Group,  Inc. 
Program  Analysers,  Ltd. 

Programart  Corp. 

Prometheus  Products 
Proprietary  Software  Systems,  Inc. 


Pure  Software,  Inc. 

QED  Software  Inc. 

QSM  Associates,  Inc. 

Qual  Trak  Corp. 

Quality  Engineering  Software,  Inc. 
Quantasm  Corp. 

Quibus  Enterprises,  Inc. 

Rational  Software,  Corp. 

Rational  Systems,  Inc. 

Ready  Systems 
Reasoning  Systems,  Inc. 

Reliable  Software  Technologies  Corp. 

Research  Triangle  Institute 
S-CUBED,  Inc. 

SAIC-Dayton 
SAS  Institute,  Inc. 

SCANDURA  &  Assoc. 

Science  Applications  Int'l  Corp. 

Scopus  Technology,  hic. 

Sequel  Corp. 

SERENA,  Inc. 

SET  Laboratories,  Inc. 

Softbridge,  Inc. 

Software  Blacksmiths,  Inc. 

Software  Business  Management,  Inc. 


FORTRAN  VERIFffiR 

Bounds-Checker,  NET-Check,  NLM-Check,  Soft-ICE 

MULTI 

Crossbow 

Change  Man 

ARC SADCA 

FORCHECK,plusFORT 

SmallTalk-80 

Hiperstation,  Hiperstation  MP  (HS/MP) 

Perennial  Driver/Monitor 
InnerVue,  preVue,  preVue-X,  Xaminer 
V-TEST,  V-TIMER,  VIDEO 

FORCE,  JOVIAL  Analysis  and  Conversion  Kit  (JACK) 
SMARTcheck,  SMARTsystem 
Productivity  Manager 
TESTBED 

APMPower,  STROBE 
Acknowledge 

AdaRAID,  Code  Auditor,  JOVIAL  IPSE  (JIPSE),  PSS  Ada 

Toolset  for  ZR34325,  PSS  ADA/JOVIAL  1750A  SUPPORT 

TOOLS,  PSS  JOVIAL  Compiler,  PSS  Link  Editor,  PSS  Macro 

Assembler 

Purify,  Quantify 

Eagle 

Size  Planner 

Distributed  Defect  Tracking  System  (DDTs) 

QES/Architect,  QES/Manager 
AsmFlow  Professional 
FORWARN 

Apex,  TestMate,  VADSWorks 
Instant-C 

Real-Time  C  (RTC),  RTAda 

REFINE/Ada,  REFINE/C,  REFINE/COBOL, 

REFINE/FORTRAN,  Software  Refinery 

PISCES- Automatic  Test  Case  Generator,  PISCES-Software 

Testability  Analysis 

Architecture  Design  &  Assessment  System  (Adas) 

DAISys,  SUPRe/DAISys 
CCount,  FCount 
SAS  System 

PRODOC  re/NuSys  Workbench 

AdaReVu,  REENgineering  Environment  and  Workbench 

(REENEW) 

QualityTEAM 

ProTerm 

SyncTrac 

PC-METRIC,  UX-METRIC 
Automated  Test  Facility  (ATF) 

C-CALL,  C-DOC,  C-METRIC,  C-REF 
DecisionVision  1  (DVl) 
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Software  Eclectics,  Inc. 

Software  Eng.  &  Enhancement  Center,  Inc. 
Software  Engineering  Assoc. 

Software  Interfaces,  Inc. 

Software  One,  Ltd. 

Software  Productivity  Research,  Inc. 

Software  Quality  Automation 
Software  Research,  Inc. 

Software  Systems  Design,  Inc. 


Software  Technology  Support  Center 
Step  Engineering 
Stepstone  Corp. 

Sterling  Software,  Inc. 

Stony  Brook  Software 
SYSCORPIntl. 

System  Management  Software,  hic. 
Systemware  Laboratories,  Inc. 

TA  Consultancy  Services  Ltd. 
Taumetric  Corp. 

Teledyne  Brown  Engineering 

Telesoft 

Testwell  Oy 

Texas  Instruments 

The  Periscope  Co.,  Inc. 

The  Software  Edge  Inc. 

Tiburon  Systems,  Inc. 

Touchstone  Software  Corp. 

Trinzic  Corp. 

Unisys 

Vector  Engineering 
Venue 

Veriiog,  Inc. 

VERITAS  Software 
Vermont  Creative  Software 
ViaSoft,  Inc. 


Visible  Systems  Corp. 
Vitro  Corp. 
WATCOM 
XA  Systems  Corp. 


SE/One 

COBOL  Analyst 

JOVIAL  Source  Code  Review  and  Metrics  (J-SCRAM),  JOVIAL 
System  Documentor  (JSD) 

SQLASSIST 
Software  One  Exchange 

Checkpoint,  SPQR  SIZER/FP  Function  Point  Calculator 
(SIZER/FP),  SPQR/20  Estimator  (SPQR/20) 

SQA  TeamTest 

STW/Advisor,  STW/Coverage,  STW/Regression 
Ada  Design  and  Documentation  Language  (ADADL), 
BugFinder/Ada,  C  Design  and  Documentation  Language 
(CDADL),  DesignGen,  Document  Generator  (DocGen), 
FORTRAN  Reverse  Eng  &  Document  System  (FREDoc), 
GrafBrowse,  QualGen,  TestGen 
JOVIAL  Reverse  Engineering  Toolset  (JRETS) 

Eclipse  29K,  Excell  930,  Express  960 
Objective-C 

ANSWER:Testpro  for  DOS,  ANSWERrTestpro  for  Windows, 

COMPAREX 

Pascal+ 

MicroSTEP 

Dynamic  Load  Balancer  (DLB) 

DOSSIER  BROWSE,  DOSSIER  PROVE 
MALPAS 

Pascal-2  Development  System 

TAGS  Case2 

TeleGen2  Ada 

TBGEN,  TCMON 

NightTrace 

WinScope 

Defect  Control  System  (DCS) 

FERRETT 

Checkit  LAN,  Checkit  PROrAnalyst 
Application  Testing  Facility  (ATF) 

Online  Session  Capture  and  Replay  (OSCAR),  UNISET 
AdaCAST 

Lisp  Object-Oriented  Programming  System  (LOOPS),  Medley 
Development  Environment  (Medley) 

Logiscope,  Logiscope  Graphical  Editor 

VISTA,  VistaKERNEL,  VislaREPLAY,  VistaTEST 

Dr.  Taylor’s  Test  (Ghost),  Vermont  HighTest 

ESW  Code  Change,  ESW  Profile  Analysis  (VIA/Recap),  ESW 

Testing,  Existing  Systems  Workbench  (ESW),  VIA/ALLIANCE, 

VIA/Insight,  VIA/Renaissance,  VIA/SmartDoc,  VIA/SmartEdit, 

VIA/SmartTest 

Visible  Analyst  Workbench  (VAW) 

Vitro  Automated  Structured  Testing  Tool  (VASTT) 

Watcom  C  32  for  DOS 
DATATEC-DS 
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Assertion  Analyzer 

Call  the  STSC  for  a  current  list  of  assertion  analyzers. 

Ada  Analyzer 

Auditor 

CONFIGURE 

Q/ARTISAN  PC 

AdaQuest 

DATRIX 

Q/AUDITOR  PL/I 

AdaReVu 

DECset 

Q/AUDITOR  PL/I  PC 

ADW/Inspector 

DOSSIER  PROVE 

QA  Administer 

ADW/Pinpoint 

Enforcer  I 

QAC 

Automated  Documentation  System 

Enforcer  II 

QAC++ 

(ADS) 

ES  REA^ision 

QA  FORTRAN 

AutoTester  For  OS/2 

esVS 

QualGen 

AutoTester  for  Windows 

File  Edit  Utility  (FEU) 

REFINE/Ada 

AutoTester  Plus 

FORCHECK 

REFINE/C 

CA-PAN/LCM 

FORTRAN  Reverse  Eng  & 

REFINE/COBOL 

CANTATA 

Document  System  (FREDoc) 

REFINE/FORTRAN 

CMS-2  Standards  Checker 

J73  Automated  Verification  System 

SCAN/COBOL 

(STDCK) 

(J73AVS) 

SEEK 

COBOL  Analyst 

lint 

Software  Refinery 

COBOL  STANDARDS 

lint-PLUS 

VAX  Source  Code  Analyzer 

ANALYZER 

Logiscope 

(SCA) 

Code  Auditor 

Panorama  C-H-/00-Analyzer 

VAXset/Program  Design 

CodeCheck 

Panorama  C++/00-SQA 

Facility  (VAXset/PDF) 

COMPAREX 

Q/Artisan 

X-ANALYSIS 

Acknowledge 

Capture-Replay  Tool 

Dr.  Taylor’s  Test  (Ghost) 

STW/Regression 

Ad/Vantage.DM* 

Evaluator 

SVS  Automated  Software 

ANSWER:Testpro  for  DOS 

FERRETT 

Testing 

ANSWER:Teslpro  for  Windows 

Hiperstation 

TestBench.DM* 

Application  Testing  Facility  (ATF) 

Hiperstation  MP  (HS/MP) 

TestMate 

Automated  Software  Test  Facility 

LoadRunner 

TestRunner 

Automated  Test  Facility  (ATI^ 

Micro  Focus  Cobol/2  Workbench 

UNISET 

Automator  QA 

Micro  Focus  Toolbox  for  UNIX 

V-TEST 

Automator  QA  (Windows  Edition) 

Micro  Focus  Workbench 

VAXset/Program  Design 

Automator  QA  with  Navigator 

Microsoft  Test 

Facility  (V  AXset/PDF) 

AutoTester 

Online  Session  Capture  and  Replay 

Vermont  HighTest 

AutoTester  for  Windows 

(OSCAR) 

VIA/SmartTest 

AutoTester  Plus 

PLAYBACK 

VIDEO 

CA-TRAPS 

preVue 

VistaREPLAY 

CA-VERIFY 

preVue-X 

WinRunner 

CarbonCopy 

ProTerm 

Workstation  Interactive  Test 

Compuware  Solution  Set 

QES/Aichitect 

Tool  (WITT) 

CPR 

QUARTZ  Remote  Terminal  Emulator 

Xaminer 

DEC/Test  Manager  (DTM) 

RealTime  Testware  (RT) 

XpertRule 

DECset 

Remote  Terminal  Emulator  (RTE) 

XRunner 

Design  Recovery  Series 

SQA  TeamTest 
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Comparator 

CA-PAN/LCM 

DocuComp 

SCONS 

CA-VERIFY 

DOSSIER  PROVE 

SHOWDIFF 

Change  Man 

dxdiff 

Source  Program  Compare 

COMPARE 

ENDEVOR 

SQA  TeamTest 

COMPAREX 

HARMONIZER 

STW/Regression 

DataBasic  II 

LOOKAT 

V-TEST 

DELTA 

Microsoft  Test 

diff  in  Op  Sys 

SCOMPARE 

Complexity  Measurer 

Ada  Analyzer 

Enforcer  II 

Panorama  C++/00-Browser 

Ada  Design  and  Documentation 

Ensemble 

Panorama 

Language  (ADADL) 

Environment  for  Code 

C++/00-Diagrammer 

Ada  Measurement  and  Analysis 

Re-Engineering  (ENCORE) 

Panorama  C++/00-SQA 

Tool  (AdaMAT) 

ES  REA^ision 

PATHVU 

Ada  Software  Development 

Excelerator  for  Design  Recovery 

PC-METRIC 

Toolset 

FORCE 

PRODOC  re/NuSys  Workbencl 

AdaQuest 

GrafBrowse 

Productivity  Manager 

AdaTEST 

Hindsight 

PSS  JOVIAL  Compiler 

ADW/Inspector 

J73  Automated  Verification  System 

Q/AUDITOR  PL/I 

Amadeus 

(J73AVS) 

Q/AUDITOR  PL/I  PC 

ARC  SADCA 

JOVIAL  Analysis  and  Conversion 

QAC 

Auto  Analyzer 

Kit  (JACK) 

QAC++ 

Battlemap  Analysis  Tool  (BAT) 

JOVIAL  Reverse  Engineering  Toolset 

QA  FORTRAN 

C  Design  and  Documentation 

(JRETS) 

QualityFirst  C 

Language  (CDADL) 

JOVIAL  Source  Code  Review  and 

Software  Quality  Analysis 

C-DOC 

Metrics  (J-SCRAM) 

Module 

C-METRIC 

Legacy  Workbench 

Source/RE 

CA-FPXpert 

lint-PLUS 

Structure  &  Logic  Analysis 

CA-METRICS 

Logiscope 

Module 

CANTATA 

MALPAS 

STW/Advisor 

CCount 

Metrics  Analysis  and  Reporting 

TCMON 

COBOL  Analyst 

System  (MARS) 

TESTBED 

COBOL/METRICS 

Metrics  Generator  (METRC) 

UNISET 

CodeCheck 

Metrics4C 

UX-METRIC 

Complexity  Measures  Tool  (CMT) 

Metrics4Fortran 

VISTA 

DATRIX 

Metrics4PascaI 

VistaKERNEL 

DecisionVision  1  (DVl) 

Panorama  C 

VistaTEST 

DESIGN  GENERATOR 

Panorama  C++ 

Vitro  Automated  Structured 

Enforcer  I 

Panorama  C++/00-Analyzer 

Testing  Tool  (VASTT) 

Coverage/Frequency  Analyzer 

AdaQuest 

Analyzer 

CA-OPTIMEER 

AdaRAID 

Auto  Analyzer 

CA-Realia  II  Workbench 

AdaTEST 

AutoFlow 

CANTATA 

AdaTune 

Basis  Branch  Analyzer  (BBA) 

cflow 

ADW/Pinpoint 

C-Cover 
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Coverage/Frequency  Analyzer  (continued) 


CMS-2  Test  Coverage  Analyzer 
(TCA) 

Computer  Tester  Analyzer 
Controller  (C-TAQ 
DCDIII 
DECset 
Ensemble 
Hindsight 

HP  64(X)0-UX  Micro.  Software 
Dev. 

HP  AxAda  Programming  Support 
Environment 
HP  Branch  Validator 
J73  Automated  Verification 
System  (J73AVS) 

Logiscope 

McCabe  Instrumentation  Tool 
Micro  Focus  COBOL/2  for  UNIX 
Micro  Focus  Cobol/2  Workbench 
Micro  Focus  Toolbox  for  UNIX 


Micro  Focus  Workbench 
Microtec  Cross  Development  Tools 
Panorama  C 
Panorama  C++ 

Panorama  C++/00-Test 
Performance  Coverage  Analyzer 
(PCA) 

PISCES-Automatic  Test  Case 
Generator 

PISCES-Software  Testability 
Analysis 
PLASMA 

PRODOC  re/NuSys  Workbench 
PSS  Ada  Toolset  for  ZR34325 
QualityFirst  C 
RXVP80 

Safe  C  Runtime  Analyzer 

SLEUTH 

Slice  Tool 

Software  Analysis  Test  Tool 


ABSTRACT/PROBE 
Ada  Design  and  Documentation 
Language  (ADADL) 

Ada  Software  Development 
Toolset 
ADAXREF 
Apex 

Application  Browser 
AsmFlow  Professional 
Automated  Documentation  System 
(ADS) 

C  Design  and  Documentation 
Language  (CDADL) 

C-CALL 

C-DOC 

C-REF 

C-SCAPE 

C-Vision  for  C 

C/ANALYST 

C3Ada  Symbolic  Debugger 

CA-COBOLVISION/Analyzer 

COBOL  Analyst 

COBOL  Glossary 

COBOL  Magic 

CodeCenter 

Crossbow 


Cross  Referencing  Tool 

cxref 
DAISys 
DATATEC-DS 
DATRIX 
DCD  III 
DEC  FUSE 

DEC  FUSE  Cross-Reference 
DECset 

Document  Generator  (DocGen) 

Documentation-Aid,  MVS  (Doc-Aid) 

DOSSIER  BROWSE 

DOSSIER  PROVE 

Eclipse  29K 

Ensemble 

Excell  930 

Express  960 

F-SCAN 

FORCE 

FORCHECK 

FORWARN 

GPSA 

GrafBrowse 

Hindsight 

HP  Ada/300  Development 
Instant-C 


STW/Coverage 
SVS  Automated  Software 
Testing 
TCMON 

Test  Coverage  Analysis  Module 

TESTBED 

TestCenter 

TestGen 

TestMate 

TREESOFT 

UNISET 

VAXsel/Program  Design 
Facility  (VAXset/PDF) 
VIA/SmartTest 
VISTA 

VistaKERNEL 

VistaTEST 

XDC 


InterCASE  KnowledgeWare 
Gateway  (IKG) 
JCL/Convert 
JCL/Cross-Reference 
JOVIAL  Analysis  and 
Conversion  Kit  (JACK) 
JOVIAL  IPSE  (JIPSE) 
JOVIAL  System  Documentor 
(JSD) 

Legacy  Workbench 
Lisp  Object-Oriented 
Programming  System 
(LOOPS) 

McCabe  Instrumentation  Tool 
Medley  Development 
Environment  (Medley) 
Navigator 
ObjectCenter 
Objective-C 
PACRE VERSE 
Panorama  C 
Panorama  C++ 

Panorama  C++/00-Analyzer 
Panorama  C++/00-Browser 
PATHVU 
PLASMA 
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Cross  Referencer  (continued) 


PRODOC  re/NuSys  Workbench 

SmallTalk-80 

VAX  Language  Sensitive 

PSS  Ada  Toolset  for  ZR34325 

SMARTcheck 

Editor  (LSE) 

PSS  Link  Editor 

Source  Code  Analyzer  (SCA) 

VAX  Source  Code  Analyzer 

PSS  Macro  Assembler 

Source/RE 

(SCA) 

QualityFirst  C 

Structure  &  Logic  Analysis  Module 

VIA/ALLIANCE 

Real-Time  C  (RTC) 

superCASE  (SCI) 

Visual  COBOL 

REENgineering  Environment  and 

SUPRe/DAISys 

Vitro  Automated  Structured 

Workbench  (REENEW) 

TeleGen2  Ada 

Testing  Tool  (VASTT) 

RTAda 

TESTBED 

X-ANALYSIS 

RXVP80 

TREESOFT 

XREFPLUS 

Data  Reducer  and  Analyzer 

Call  the  STSC  for  a  current  list  of  data  reducers  and  analyzers. 

Data  Extractor 

Call  the  STSC  for  a  current  list  of  data  extractors. 

Debugger 

Call  the  STSC  for  a  current  list  of  debuggers. 

Defect/Change  Tracker 

Ada  Analyzer 

DecisionVision  1  (DVl) 

QAC 

Ada  Measurement  and  Analysis 

Defect  Control  System  (DCS) 

QAC++ 

Tool  (AdaMAT) 

Distributed  Defect  Tracking  System 

QA  FORTRAN 

AdaReVu 

(DDTs) 

QA  Process  Control  Manager 

Amadeus 

DOSSIER  PROVE 

(QAPCM) 

AutoTester  Plus 

Dr.  Taylor’s  Test  (Ghost) 

QA/S 

BugBase 

FOR.STUDY 

QualGen 

BugFinder/Ada 

Legacy  Workbench 

QualityTEAM 

C  Design  and  Documentation 

Metrics  Analysis  and  Reporting 

SEER 

Language  (CDADL) 

System  (MARS) 

Software  Quality  Analysis 

C-DOC 

Metrics4Project 

Module 

CA-METRICS 

MicroSTEP 

SQA  TeamTest 

CA-OPTIMIZER 

Online  Session  Capture  and  Replay 

SyncTrac 

CaseQMS 

(OSCAR) 

TREESOFT 

COHESION  Team/SEE 

Problem  Alert  System  (PAS) 

Vermont  HighTest 

CustomQA 

QA  Administer 

Vitro  Automated  Structured 

Testing  Tool  (VASTT) 

Emulator 


Call  the  STSC  for  a  current  list  of  emulators. 
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Network  Analyzer 

Call  the  STSC  for  a  current  list  of  network  analyzers. 

Ada  Analyzer 

Performance/Timing  Analyzer 

InnerVue 

Remote  Terminal 

AdaRABD 

LoadRunner 

Emulator(RTE)Safe  C 

AdaTEST 

Micro  Focus  COBOL/2  for  UNIX 

Runtime  Analyzer 

AdaTune 

Micro  Focus  Toolbox  for  UNIX 

SAS  System 

ADAXPA 

Micro  Focus  Workbench 

SimCASE  - 

ADW/Pinpoint 

Microsoft  Source  Profiler 

Simulator/Debugger 

Allegro  Composer  Development 

Microtec  Cross  Development  Tools 

(SimCASE) 

Environment 

MICS  MVS  Performance  Manager 

SmallTalk-80 

AnacatAPMPower 

Monitor/Plus 

STATUS 

Auto  Analyzer 

NighUrace 

STROBE 

Automated  Test  Facility  (ATF) 

Panorama  C++ 

TD2000 

AutoTester  for  Windows 

Pascal+ 

TestPoint 

AutoTester  Plus 

Pascal-2  Development  System 

time 

BBN/Catalyst  Software 

perfmeter 

TrackDeck 

Business  BenchMark 

perfmon 

TREESOFT 

CA-OPTIMIZER 

Performance  Coverage  Analyzer 

V-TEST 

Checkit  PRO:Analyst 

(PCA) 

V-TIMER 

CLAS  2000 

pixie 

Variable  Value  Monitor 

CRYSTAL 

preVue 

(VVM-1) 

DECset 

preVue-X 

VAX  Performance  Advisor 

Dr.  Taylor’s  Test  (Ghost) 

PROBLEM  PROGRAM 

(VPA) 

Dynamic  Load  Balancer  (DLB) 

EVALUATION  (PPE) 

VAXset/Program  Design 

Eclipse  29K 

PSS  ADA/JOVIAL  1750A 

Facility  (VAXset/PDF) 

Express  960 

SUPPORT  TOOLS 

Vermont  HighTest 

Fortran  Utility  System 

QAPlus 

Viewpoint 

FPT 

Quantify 

vtimes 

Hindsight 

QUARTZ  Remote  Terminal  Emulator 

Watcom  C  32  for  DOS 

HP  AxAda  Programming  Support 

QUEMAN 

WinScope 

Environment 

Xaminer 

Ada  Type  Interchange  Generator 
DesignGen 

Ensemble 

Requirements-Based  Test  Case  Generator 

QES/Architect 

Requirements  and  Traceability 
Management  (RTM) 

SOFTTEST 

StP/T 

T 

Teamwork/TestCase 

Run-Time  Error  Checker 

ADAXPA 

Bounds-Checker 

CLAS  2000 

Advanced  Debugging  System 

CA-EZTEST/CICS 

MicroSTEP 

(ADS) 

CA-lnterTest 

NET-Check 

Automated  Documentation  System 

CA-OPTIMIZER 

NLM-Check 

(ADS) 

Checkit  LAN 

Pascal+ 
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Run-Time  Error  Checker  (continued) 

PLASMA 

Purify 

QAPlus 

Quantify 

Run  Time  Debugger  TestPoint 

Safe  C  Runtime  Analyzer  TestRunner 

Soft-ICE  VADSWorks 

TestCenter  VIA/SmartTest 

Simulator 

Call  the  STSC  for  a  current  list  of  simulators. 

Size  Measurer 

Ada  Measurement  and  Analysis 

Complexity  Measures  Tool  (CMT) 

Metrics  Manager  (M  M) 

Tool  (AdaMAT) 

DATRIX 

Metrics4C 

Ada  Software  Development 

DecisionVision  1  (DVl) 

Metrics4Fortran 

Toolset 

Enforcer  I 

Metrics4Pascal 

AdaQuest 

Enforcer  11 

Panorama  C 

AdaTEST 

ES  REA^ision 

Panorama  C-h- 

ADW/Inspector 

ESW  Profile  Analysis  (VIA/Recap) 

PC-METRIC 

Amadeus 

Excelerator  for  Design  Recovery 

Productivity  Manager 

ARC  SADCA 

FCount 

QualGen 

Auto  Analyzer 

FIRSTCASE 

SEER 

Battlemap  Analysis  Tool  (BAT) 

GrafBrowse 

size 

C  Design  and  Documentation 

Hindsight 

Size  Planner 

Language  (CDADL) 

J73  Automated  Verification  System 

Software  Quality  Analysis 

C-DOC 

(J73AVS) 

Module 

C-METRIC 

JOVIAL  Analysis  and  Conversion 

SPQR  SIZER/FP  Function 

CA-FPXpert 

Kit  (JACK) 

Point  Calculator  (SIZER/FP) 

CA-METRICS 

JOVIAL  Source  Code  Review  and 

SPQR/20  Estimator  (SPQR/20) 

CANTATA 

Metrics  (J -SCRAM) 

STW/Advisor 

CCount 

Legacy  Workbench 

UX-METRIC 

Checkpoint 

Logiscope 

Vitro  Automated  Structured 

COBOL  Analyst 

Metrics  Analysis  and  Reporting 

Testing  Tool  (VASTT) 

COBOL/METRICS 

System  (MARS) 

CodeCheck 

Metrics  Generator  (METRC) 

Status  Displayer/Session  Documenter 

Call  the  STSC  for  a  current  list  of  status  display ers/session  documenters. 

Structure  Checker 

Ada  Analyzer 

Apex 

AsmFlow  Professional 

Ada  Software  Development 

Application  Browser 

AutoAnalyzer 

Toolset 

ARC  SADCA 

AutoDiagrammer 

AdaProbe/ICE 

Architecture  Design  &  Assessment 

AutoFlow 

AdaQuest 

System  (Adas) 

A-175 


Software  Technology  Support  Center 


Automated  Documentation  System 
(ADS) 

AutoStructureChart 
Battlemap  Analysis  Tool 
(BAT)BugFinder/Ada 
C  Design  and  Documentation 
Language  (CDADL) 

C  Source  Analyzer 
C-CALL 
C-DOC 
C-Vision  for  C 

CA-COBOLVISION/Analyzer 

CA-OPTIMIZER 

CANTATA 

CARDtools  (CARDtools) 

CLEAR  Plus 

COBOL  Analyst 

COBOL  Magic 

COBOL/METRICS 

CodeBreaker 

CONVERSION  ENGINE 

CPR 

ctrace 

DCD  III 

DEC  FUSE 

DEC  FUSE  Call  Graph  Browser 
DecisionVision  1  (DVl) 
Documentation-Aid,  MVS 
(Doc-Aid) 

Eagle 
Enforcer  I 
Enforcer  II 
Ensemble 
ES  RE/Vision 
ESW  Testing 

Excelerator  for  Design  Recovery 
Excell  930 

Existing  Systems  Workbench 
Expert  Debugging  Software 
Assistant  (EDSA) 

F-SCAN 

FORCE 

FORCHECK 


Structure  Checker  (continued) 

FORTRAN  Reverse  Eng  & 

Document  System  (FREDoc) 
FORWARN 
FOR.STRUCT 
Foundation  VistalO 
gprof 
Hindsight 
Insight 
IntegrAda 

J73  Automated  Verification  System 
(J73AVS) 

JOVIAL  IPSE  (JIPSE) 

JOVIAL  Reverse  Engineering  Toolset 
(JRETS) 

JOVIAL  System  Documentor  (JSD) 

Legacy  Workbench 

lint 

Lisp  Object-Oriented  Programming 
System  (LOOPS) 

Logiscope 

Logiscope  Graphical  Editor 
MALPAS 

McCabe  Instrumentation  Tool 
McCabe  Slice  Tool 
Medley  Development  Environment 
(Medley) 

Metrics  Generator  (METRC) 

Micro  Focus  Cobol/2  Workbench 

Micro  Focus  Toolbox  for  UNIX 

Microtec  Cross  Development  Tools 

MULTI 

ObjectCenter 

Objective-C 

Open  ACCLIM8 

PACRE VERSE 

Panorama  C 

Panorama  C++ 

Panorama  C++/00-Browser 
Panorama  C++/00-Diagrammer 
PATHVU 
PLASMA 

PRODOC  re/NuSys  Workbench 
Q/AUDITOR  PL/I 


Q/AUDITOR  PL/I  PC 

QAC 

QAC++ 

QA  FORTRAN 
QualityFirst  C 
RE-DOC 
RE-SPEC 

REENgineering  Environment 
and  Workbench  (REENEW) 
REFESE/Ada 
REFINE/C 
REFINE/COBOL 
REFINE/FORTRAN 
RXVP80 
SCAN/COBOL 
SE/One 
SEER 
Slice  Tool 

Software  One  Exchange 
Software  through  Pictures  (StP) 
Source/RE 

Structure  &  Logic  Analysis 
Module 

STW/Coverage 

Teamwork/Ada 

TESTBED 

UX-METRIC 

VAX  Source  Code  Analyzer 
(SCA) 

VIA/ALLIANCE 
VI  A/Insight 
VIA/Renaissance 
VIA/SmartDoc 
VIA/SmartEdit 
Visible  Analyst  Workbench 
(VAW) 

VS  FORTRAN 
Complier/Library/Debug 
X-ANALYSIS 
XAda 

XRAY  Source  Explorer 


Syntax  &  Semantics  Analyzer 

AdaVerify 
ADW/Pinpoint 
AsmFlow  Professional 


ACCENT  R  ADANON 

Ada  Measurement  and  Analysis  AdaReVu 

Tool  (AdaMAT)  AdaTEST 
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Auto  V&V/Ada 

Syntax  &  Semantics  Analyzer  (continued) 

FORTRAN  VERIFIER 

QA  C++ 

C->it  (see.it) 

FORTRAN-Unt 

QA  FORTRAN 

C/ANALYST 

FORWARN 

QualGen 

C3Ada  Symbolic  Debugger 

GPSA 

REFINE/Ada 

CA-OPTIMIZER 

Hindsight 

REFINE/C 

CANTATA 

HP  Softbench 

REFINE/COBOL 

CMS-2  Design  Analyzer 

Instant-C 

REFINE/FORTRAN 

(DESAN) 

Intck  C++ 

RXVP80 

COBOL  Analyst 

InterCASE  KnowledgeWare  Gateway 

SCAN/COBOL 

COBOL  STANDARDS 

JOVIAL  Source  Code  Review  and 

SE/One 

ANALYZER 

Metrics  (J-SCRAM) 

SMARTcheck 

COBOL-lint 

JOVIAL  System  Documentor 

SMARTsystem 

CodeCheck 

lint 

Software  One  Exchange 

CONVERSION  ENGINE 

lint-PLUS 

SQLASSIST 

DATATEC 

MALPAS 

Static  Analysis  Tool  Set 

DCDIII 

MAX  Macro  Assembler  (MAX) 

(SATS) 

DEC  FUSE 

Open  ACCLIM8 

STW/Advisor 

DECset 

Panorama  C 

TAGS  Case2 

DOSSIER  BROWSE 

Panorama  C++ 

TESTBED 

ESW  Code  Change 

Panorama  C++/00-Analyzer 

VAX  Language  Sensitive 

Excelerator  for  Design  Recovery 

PC-Lint 

Editor  (LSE) 

Expert  Debugging  Software 

PCYACC 

VAX  Source  Code  Analyzer 

Assistant  (EDSA) 

PLASMA 

(SCA) 

F-SCAN 

plusFORT 

VIA/SmarfEdit 

FlexeLint 

PRODOC  re/NuSys  Workbench 

VS  FORTRAN 

FORCE 

prof 

Complier/Library/Debug 

FORCHECK 

QAC 

XAda 

Test  Data  Generator 

Call  the  STSC  for  a  current  list  of  test  data  generators. 

Test  Execution  Manager 

AccuTest 

Automator  QA  with  Navigator 

CustomQA 

AdA^antage.DM* 

AutoTester 

DEC/Test  Manager  (DTM) 

Ada  Design  and  Documentation 

AutoTester  For  OS/2 

DECset 

Language  (ADADL) 

AutoTester  for  Windows 

Dr.  Taylor’s  Test  (Ghost) 

AdaCAST 

AutoTester  Plus 

Evaluator 

AdaTEST 

Business  BenchMark 

FERRETT 

Anacat 

BUSTER  Test  Management  System 

Hiperstation 

ANSWER:Testpro  for  DOS 

CA-TRAPS 

Hiperstation  MP  (HS/MP) 

Application  Testing  Facility  (ATF) 

CA-VERIFY 

LoadRunner 

Automated  Software  Test  Facility 

CANTATA 

Metrics4C 

Automated  Test  Facility  (ATF) 

Compuware  Solution  Set 

Metrics4Fortran 

Automator  QA 

CPR 

Micro  Focus  Cobol/2 

Automator  QA  (Windows  Edition) 

Cross  System  Product  (CSP) 

Workbench 
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Test  Execution  Manager  (Continued) 

Micro  Focus  Workbench 

Software  Analysis  Test  Tool 

TREESOFT 

Perennial  Driver/Monitor 

SQA  TeamTest 

UNISET 

PISCES-Software  Testability 

STW/Regression 

V-TEST 

Analysis 

TBGEN 

VAXset/Program  Design 

PLAYBACK 

Test/Cycle 

Facility  (VAXset/PDF) 

QES/Manager 

TESTBED 

Vermont  HighTest 

RealTime  Testware  (RT) 

TestBench.DM* 

VIA/SmartTest 

Remote  Terminal  Emulator  (RTE)  TestCenter 

Visual  COBOL 

SEDIT.DB 

TestGen 

WinRunner 

SEDIT.MVS 

TestMate 

Workstation  Interactive  Test 

SOFTTEST 

TestRunner 

Tool  (WITT) 

XRunner 

Test  Planner 

Call  the  STSC  for  a  current  list  of  test  planners. 

Validation  Suite 


Call  the  STSC  for  a  current  list  of  validation  suites. 
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In  the  December  1992  issue  of  CrossTalk,  the  STSC  published  "Product  Critiques 
Revisited,"  which  discusses  the  STSC's  Product  Critique  System.  Sections  B.l  through  B.3  are  a 
reprint  of  that  article.  Section  B.3  contains  a  blank  product  critique.  Completed  product  critiques 
for  a  number  of  software  tools  can  be  obtained  by  contacting  the  STSC. 

B.1  Product  Critique  Concepts 

Selecting  the  right  software  products  to  improve  your  software  development  process  can 
incorporate  hundreds  of  criteria.  We  use  the  term  technology  in  the  broad  sense  to  include  processes, 
methods,  techniques,  tools,  tool  sets,  and  environments.  It  is  easy  for  engineers  to  get  wrapped  up 
in  detailed  evaluations  of  software  products  by  listing  criteria,  setting  up  an  evaluation  matrix, 
developing  a  scoring  scheme,  determining  weighting  factors,  and  so  forth.  This  is  not  wrong;  detailed 
evaluations  are  essential  once  your  options  have  been  narrowed  down  to  a  manageable  size.  Yet,  we 
often  overlook  the  value  of  a  less  quantitative  approach  of  collecting  testimonials  or  critiques  from 
experienced  users  to  narrow  those  options. 

When  you  buy  a  car,  it  is  useful  to  talk  with  friends,  relatives,  and  neighbors  who  own  the 
same  models  you  are  considering.  You  want  to  leverage  off  both  their  good  and  bad  experiences  to 
narrow  your  choices  and  reaffirm  your  selection.  Once  your  choices  have  been  reduced,  you  can  use 
evaluations,  specifications,  additional  options,  and  test  drives  to  select  a  car  to  buy. 

Selecting  a  software  technology  is  no  different,  except  that  most  friends,  relatives,  and 
neighbors  lack  the  experience  you  need.  The  STSC  is  facilitating  the  exchange  of  tool  experiences 
with  the  product  critique  form,  which  you  will  find  in  Section  B.3.  If  you  are  an  experienced  user  of 
a  particular  software  product,  please  take  a  few  minutes  to  share  your  experience  by  filling  in  this 
form  and  mailing  or  faxing  it  to  the  STSC.  Make  copies  of  the  blank  form  as  needed  for  each  product 
to  be  critiqued. 

The  goals  of  the  STSC  Product  Critique  System  are 

•  To  convey  actual  experiences  of  product  users  to  potential  product  buyers. 

•  To  help  consumers  of  software  technology  reduce  their  candidate  lists  effectively  and 
at  low  cost. 

•  To  provide  software  product  developers  with  constructive  criticism  from  users  about 
the  strengths  and  weaknesses  of  their  products,  particularly  requirements  analysis  and 
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design,  testing,  reengineering,  documentation,  reuse,  project  management,  project 
estimation,  and  software  engineering  environment  products. 

•  To  complement  or  summarize  quantitative  evaluations  for  use  in  the  final  selection  of 
software  technologies. 

Note  that  the  STSC  emphasizes  the  importance  of  selecting  a  sound  software  development 
or  maintenance  methodology  before  selecting  a  software  product  to  automate  that  methodology. 
Some  products  have  a  built-in  methodology  that  will  require  some  user  training  before  the  tool  can 
be  used  effectively. 

B.2  Product  Critique  Instructions 

Every  entry  of  the  product  critique  should  be  brief  and  basically  cover  the  issues  of  ease  of 
use,  power,  robustness,  functionality,  ease  of  insertion,  and  quality  of  vendor  support  [FIRT87].  The 
intent  is  to  capture  your  impressions  of  the  product  rather  than  have  you  list  the  tool's  features.  Most 
of  the  blocks  and  fields  are  self-explanatory,  but  some  additional  explanations  on  the  following  fields 
may  be  helpful. 

a.  Operating  Environment.  Describe  the  main  components  (hardware  and  software)  that 
make  up  the  system,  particularly  those  parts  of  which  the  product  is  dependent  for  proper 
utilization.  Include  memory  and  disk  requirements. 

b.  Project  Description.  Describe  unclassified  details  about  the  project  that  would  help  product 
critique  readers  to  better  understand  the  technology's  application  in  your  environment. 

c.  Keep  name  and  company  confidential.  Circle  "Y"  or  "N"  to  indicate  whether  or  not  you 
would  be  willing  to  have  other  potential  tool  buyers  or  vendors  call  you  for  further 
information. 

d.  Will  you  use  this  product  on  your  next  project?  Do  yon  prefer  another  product?  These 
are  essentially  bottom-line  questions  about  your  opinion  of  the  technology.  Enter  supporting 
information  in  the  blocks  for  strengths  and  weaknesses.  Enter  the  name  of  the  preferred 
product  in  the  advice  block. 
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e.  Notable  Strengths  of  this  Product.  Describe  notable  features  and  performance 
characteristics  that  make  the  technology  indispensable  or  better  than  other  technologies  with 
similar  features.  Cite  specific  examples  that  helped  you  form  your  impressions. 

f.  Notable  Weaknesses  of  this  Product.  Describe  annoying  problems  or  deficiencies  that 
reduce  the  technology's  effectiveness. 

g.  Advice  for  Potential  Users  or  Buyers  of  this  Product.  This  block  provides  information 
(not  necessarily  good  or  bad)  that  is  important  for  a  potential  buyer;  for  example,  you  could 
write,  "It  is  essential  to  use  an  accelerator  card  and  a  20-inch  color  monitor." 

h.  Vendor  Comments.  The  product  reviewer  does  not  complete  this  block.  The  vendor  will 
be  given  the  opportunity  to  respond  to  the  strengths  and  weaknesses  identified  by  the 
reviewer. 

If  you  are  interested  in  using  STSC  product  critiques  to  help  you  select  software  products, 
please  contact  the  STSC  directly. 

B.3  Blank  Product  Critique 

The  next  page  contains  a  blank  product  critique  form.  Make  copies  of  this  form  as  needed 
for  product  critiques. 
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(This  page  has  been  intentionally  left  blank.) 
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Sponsor  /  Phone 

ACM  SIGAda 

510-814-1775/719-590-5224 

ACM  SIGSOFT 

4 1 3-545-2742  /  3 1 0-822- 1511 

Advanced  Information  Technologies 
800-882-8684 

AT&T  Lab 

800-282-2828 

Bender  &  Associates 

415-924-9196 

Berard  Software  Engineering 
301-417-9884 

Computer  Resources  &  Engineering  Office 
205-955-1995 

Data-Tech  Institute 

201-478-5400  ext  229 

Digital  Consulting,  Inc. 

508-470-3880 

Digital  Consulting,  Inc. 

508-470-3880 

Digital  Consulting,  Inc. 

508-470-3880 

Digital  Consulting  Inc. 

508-470-3880 

Educational  Foundation  Data  Processing 
Management  Association  (EFDPMA) 
201-284-2946 

Educational  FoundatioifData  Processing 
Management  Association  (EFDPMA) 
201-284-2946 

Fastrak  Training  Inc. 

800-488-2321 

George  Washington  University 

408-265-1960  /  800-424-9773 

George  Washington  University 

408-265-1960  /  800-424-9773 


Conference 

TRI-ADA 

Annual  ACM  SIGSOFT  Symposium  on  the  Foundations  of 
Software  Engineering 

Systems  Testing  and  Quality  Assurance  Techniques 

Software  Reliability  Engineering  Application  Workshop 

CASE  WORLD  -  The  National  Application  and  Development 
Conference 

Testing  of  Object-Oriented  Software  (TOOS) 

Annual  Software  Engineering  Symposium/Test  Tools 

Improving  the  Software  Testing  Process 

Applied  Software  Measurement  (ASM)  with  Capers  Jones 

Improving  Software  Quality 

JAD:  Commitment  and  Quality  Through  Consensus 

National  Software  Reengineering  and  Maintenance  Conference 

Automated  Software  Test  and  Evaluation  Conference 

Software  Process  Improvement  &  Automation 

Software  Quality  Assurance 

Assessment  of  Quality  Software  Development  Tools  Symposium 

Software  Verification  and  Validation 
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Sponsor  /  Phone 

IEEE  Aerospace  and  Electronic  Systems  Society 
513-237-7971 

IEEE  Aerospace  and  Electronics  Systems  Society 
703-734-5611 

IEEE  Computer  Society 
919-515-7146 

IEEE  Computer  Society,  SMA,  ACM 
301-572-3815 

IEEE  Computer  Society  &  Technical  Committee 
408-265-1960 

IEEE  Computer  Society  &  Technical  Committee 
216-368-2802 

IEEE  Computer  Society  &  Technical  Committee 
919-515-7886 

IEEE  Computer  Society  &  Technical  Committee 
415-723-1451 

IEEE  Philadelphia  Section 
814-941-4666 

IEEE  Reliability  Society 
303-673-6223 

IEEE  Reliability  Society  Denver  Chapter 
303-492-8118 

International  Test  and  Evaluation  Association 
703-631-6220 

International  Test  and  Evaluation  Association 
703-631-6220 

Learning  Group  International 
800-843-8733 

McCabe  &  Associates 
800-638-6316 

National  Institute  for  Software  Quality  & 

Productivity 

301-983-3295 

Pacific  Northwest  Software  Quality  Conference 
503-223-8633 


Conference 

Automatic  Testing  Conference 

COMPASS,  Annual  Conference  on  Computer  Assurance 
Software  Reliability  Engineering 

Conference  on  Software  Maintenance 

Assessment  of  Quality  Software  Development  Tools  Symposium 

International  Conference  on  Software  Maintenance  and  Tools 
Fair 

International  Symposium  on  Software  Reliability  Engineering 
Pacific  Test  Workshop 

International  Test  Conference 

International  Software  Reliability  Engineering  Conference 

Software  Reliability  Symposium 

Annual  Conference  and  Exposition  on  Testing  Computer 
Software 

Annual  Test  Technology  Transfer  Symposium 
Software  Quality  and  Testing  Seminar 
Reverse  Engineering/Structured  Maintenance 
TQM  for  Software 

Pacific  Northwest  Software  Quality  Conference 
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Sponsor  /  Phone 

Portland  State  University 
503-690-1460 

Purdue  University,  Division  of  Conferences 
317-494-7231 

Quality  Assurance  Institute 
407-363-1111 

SIGS  Conference 

212-274-9135 

Society  for  Software  Quality 

619-297-1544,  Ext.  3005 

Software  Association  of  Oregon  (SAO) 
503-520-1977 

Software  Engineering  Institute 
412-268-7388 

Software  Engineering  Institute 
412-268-7388 

Software  Engineering  Institute 
412-268-7388 

Software  Productivity  Research 
617-273-0140 

Software  Quality  Engineering 
904-268-8639 

Software  Quality  Engineering 
800-423-8378 

Software  Quality  Engineering 
800-423-8378 

Software  Quality  Engineering 
800-423-8378 

Software  Quality  Engineering 
800-423-8378 

Software  Quality  Engineering 
800-423-8378 

Software  Quality  Engineering 
800-423-8378 

Software  Quality  Engineering 
904-268-8639 


Conference 

Annual  Oregon  Workshop  on  Software  Metrics  (AOWSM) 

Workshop  on  Issues  in  Software  Reliability  and  Testing 

International  Conference  on  Software  Testing 

Object  Expo  -  The  National  Conference  &  Exposition 

Achieving  Software  Quality,  Annual  Debate 

Northwest  Software  Engineering  Tools  Fair 

Software  Engineering  Symposium 

Software  Capability  Evaluation  Overview  Seminar 

Engineering  an  Effective  Software  Measurement  Program 

Function  Point  Analysis  Workshop  (through  Digital  Consulting) 
Applications  of  Software  Measurement 

International  Conference  on  Applications  of  Software 
Measurement  (ASM) 

European  International  Conference  on  Software  Testing, 
Analysis  &  Review  (EuroSTAR) 

Software  Quality  Engineering  Weeks 

Software  Measurement  Seminar 

Software  Testing,  Analysis  &  Review  (STAR)  Conference 
Systematic  Software  Testing  Seminar 
Reliability  Growth  Testing 
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Sponsor  /  Phone 

Software  Quality  Engineering 
904-268-8639 

Software  Quality  Engineering 
904-268-8639 

Software  Quality  Engineering 
904-268-8639 

Software  Research,  Inc. 

800-942-7638 

Software  Research,  Inc. 

800- 942-7638 

Software  Technology  Support  Center  (STSC) 

801- 777-7411 

System  Quality  Consultants,  Inc. 
619-697-0085 

System  Quality  Consultants,  Inc 
702-922-0202 

System  Quality  Consultants,  Inc. 
702-922-0202 

Technology  Transfer  Institute 
310-394-8305 

Technology  Transfer  Institute 
310-394-8305 

Univ.  California  at  Berkeley 
510-642-4111 

Univ.  California  at  Berkeley 
510-642-6117 

Univ.  of  Washington 

206-543-5539 

U.S.  Army  Test  &  Evaluation  Command 
410-278-1478 

U.S.  Naval  Post  Graduate  School 
408-656-2719 


Conference 

Software  Measurement 

Technical  Review  Inspections  (TRI) 

Test  Automation  Workshop 

Software  Test  Automation  Seminar 

International  Software  Quality  Week 

Annual  Software  Technology  Conference 

Formal  Software  Inspections 

Software  Quality  Assurance 

Software  Testing  Seminar 

Software  Inspection  Leadership 

Software  Quality  Management 

Advanced  Software  Engineering/Software  Testing 

Software  Testing  and  Quality  Assurance  Seminar 

Usability  Testing  of  the  Documentation  and  User  Interface  of 
Computer  Systems 

Test  Technical  Symposium 

International  Symposium  on  Software  Reliability  Engineering 


U.  S.  Professional  Development  Institute  Lifecycle  Software  Quality  Assurance  Seminar 

301-445-4405 
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Sponsor  /  Phone 

U.S.  Professional  Development  Institute 
301-445-4405 

U.S.  Professional  Development  Institute 
301-445-4405 

U.S.  Professional  Development  Institute 
301-445-4405 

U.S.  Professional  Development  Institute 
301-445-4405 

U.S.  Professional  Development  Institute 
301-445-4405 

Wright  Labs 

513-255-2446 


Conference 

Software  Quality  Assurance  in  Government 
Software  Testing:  An  Integrated  Approach 
Software  Testing  in  Government 
Test  Case  Design 

International  Testing  Computer  Software 
Test  Facility  Working  Group  Conference 
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Acceptance  Testing 

Algorithmic  Test  Case 
Generation 

Alpha  Testing 
APSE 

Automated  Testing 
Beta  Testing 
Black-Box  Testing 
Bottom-Up  Testing 

Boundary  Value  Analysis 

Branch  Coverage  Testing 
CASE 

CAST 

Cause-Effect  Graphing 

Clear-Box  Testing 


Formal  testing  conducted  to  determine  whether  or  not  a  system 
satisfies  its  acceptance  criteria  -  enables  a  customer  to  determine 
whether  or  not  to  accept  the  system  [YOUN89]. 

A  computational  method  for  identifying  test  cases  from  data,  logical 
relationships,  or  other  software  requirements  information. 

Testing  of  a  software  product  or  system  conducted  at  the  developer's 
site  by  the  customer  [YOUN89]. 

Ada  Programming  Support  Environment. 

That  part  of  software  testing  that  is  assisted  with  software  tool(s)  that 
does  not  require  operator  input,  analysis,  or  evaluation. 

Testing  conducted  at  one  or  more  customer  sites  by  the  end-user  of 
a  delivered  software  product  or  system  [YOUN89]. 

Functional  testing  based  on  requirements  with  no  knowledge  of  the 
internal  program  structure  or  data.  Also  known  as  closed-box  testing. 

An  integration  testing  technique  that  tests  the  low-level  components 
first  using  test  drivers  for  those  components  that  have  not  yet  been 
developed  to  call  the  low-level  components  for  test. 

A  test  data  selection  technique  in  which  values  are  chosen  to  lie  along 
data  extremes.  Boundary  values  include  maximum,  minimum,  just 
inside/outside  boundaries,  typical  values,  and  error  values  [MYER79]. 

A  test  method  satisfying  coverage  criteria  that  requires  each  decision 
point  at  each  possible  branch  to  be  executed  at  least  once  [IEEE83]. 

Computer-Aided  (or  Computer-Assisted)  Software  Engineering 
consists  of  the  automated  capability  to  implement  the  discipline  of 
software  engineering  within  a  lifecycle  phase  or  across  multiple 
lifecycle  phases. 

Computer-Aided  Software  Testing  consists  of  the  automated 
capability  to  implement  software  testing  functions  within  a  lifecycle 
phase  or  across  multiple  lifecycle  phases. 

A  testing  technique  that  aids  in  selecting,  in  a  systematic  way,  a  high- 
yield  set  of  test  cases  that  logically  relates  causes  to  effects  to  produce 
test  cases.  It  has  a  beneficial  side  effect  in  pointing  out 
incompleteness  and  ambiguities  in  specifications  [MYER79]. 

Another  term  for  white-box  testing.  Structural  testing  is  sometimes 
referred  to  as  dear-box  testing  since  "white-boxes"  are  considered 
opaque  and  do  not  really  permit  visibiUty  into  the  code.  Also  known 
as  glass-box  or  open-box  testing. 
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Cyclomatic  Complexity 
Data  Flow  Analysis 

Debugging 

Defect 

Defect  Analysis 

Defect  Density 
Desk  Checking 

DT&E 

Dynamic  Analysis 

Equivalency  Class 
.  Partitioning 

Error 

Error-Based  Testing 

Essential  Complexity 


A  measure  of  the  number  of  linearly  independent  paths  through  a 
program  module  [MCCA89]. 

Consists  of  the  graphical  analysis  of  collections  of  (sequential)  data 
definitions  and  reference  patterns  to  determine  constraints  that  can  be 
placed  on  data  values  at  various  points  of  executing  the  source 
program  [YOUN89]. 

The  act  of  attempting  to  determine  the  cause  of  the  symptoms  of 
malfunctions  detected  by  testing  or  by  frenzied  user  complaints 
[BEIZ90]. 

(1)  A  manifestation  of  an  error  made  by  a  software  developer 
[IDA92].  (2)  An  error  made  by  a  software  developer  that  can  appear 
in  code  or  documentation. 

Using  defects  as  data  for  continuous  quality  improvement.  Defect 
analysis  generally  seeks  to  classify  defects  into  categories  and  identify 
possible  causes  in  order  to  direct  process  improvement  efforts. 

Ratio  of  the  number  of  defects  to  program  length  (a  relative  number). 

A  form  of  manual  static  analysis  usually  performed  by  the  originator. 
Source  code  documentation,  etc.,  is  visually  checked  against 
requirements  and  standards  [PRIC94]. 

Developmental  Test  and  Evaluation  focuses  on  the  technological  and 
engineering  aspects  of  the  system  or  equipment  items  |PACS79]. 

The  process  of  evaluating  a  program  based  on  execution  of  that 
program  [IEEE83].  Dynamic  analysis  approaches  rely  on  executing 
a  piece  of  software  with  selected  test  data. 


A  testing  technique  that  involves  identifying  a  finite  set  of 
representative  input  values  that  invoke  as  many  different  input 
conditions  as  possible  [MYER79]. 

(1)  A  discrepancy  between  a  computed,  observed,  or  measured  value 
or  condition  and  the  true,  specified,  or  theoretically  correct  value  or 
condition  [IEEE83],  and  (2)  A  mental  mistake  made  by  a  programmer 
that  may  result  in  a  program  fault  [YOUN89]. 

Testing  where  information  about  programming  style,  error-prone 
language  constructs,  and  other  programming  knowledge  is  applied  to 
select  test  data  capable  of  detecting  faults,  either  a  specified  class  of 
faults  or  all  possible  faults  [YOUN89]. 

A  measure  of  the  level  of  "structuredness"  of  a  program  [MCCA89]. 


E-199 


Software  Technology  Support  Center 


Evaluation 

Execution 
Exhaustive  Testing 
Failure 

Failure-Directed  Testing 
Fault 

Fault-Based  Testing 

Fault  Tree  Analysis 

Formal  Review 

Functional  Testing 

Function  Points 

Heuristics  Testing 
Hybrid  Testing 

Incremental  Analysis 

Infeasible  Path 


The  process  of  examining  a  system  or  system  component  to  determine 
the  extent  to  which  specified  properties  are  present  [YOIJN89]. 

The  process  of  carrying  out  an  instruction  or  the  instructions  of  a 
computer  program  by  a  computer  [IEEE83]. 

Executing  the  program  with  all  possible  combinations  of  values  for 
program  variables  [ADRI82]. 

The  inability  of  a  system  or  system  component  to  perform  a  required 
function  within  specified  limits.  A  failure  may  be  produced  when  a 
fault  is  encountered  [IEEE83]. 

Testing  based  on  the  knowledge  of  the  types  of  errors  made  in  the 
past  that  are  likely  for  the  system  under  test. 

A  manifestation  of  an  error  in  software.  A  fault,  if  encountered,  may 
cause  a  failure  [IEEE83]. 

Testing  that  employs  a  test  data  selection  strategy  designed  to 
generate  test  data  capable  of  demonstrating  the  absence  of  a  set  of 
pre-specified  faults;  typically,  frequently  occurring  faults  [YOUN89]. 

A  form  of  safety  analysis  that  assesses  hardware  safety  to  provide 
failure  statistics  and  sensitivity  analyses  that  indicate  the  possible 
effect  of  critical  failures  [YOUN89]. 

Formal  reviews  are  technical  reviews  conducted  with  the  customer 
including  the  types  of  reviews  called  for  in  DOD-STD-2167A 
(Preliminary  Design  Review,  Critical  Design  Review,  etc.) 

Application  of  test  data  derived  fi'om  the  specified  functional 
requirements  without  regard  to  the  final  program  structure  [ADRI82]. 
Also  known  as  black-box  testing. 

A  consistent  measure  of  software  size  based  on  user  requirements. 
Data  components  include:  inputs,  outputs,  etc.  Environment 
characteristics  include:  data  communications,  performance,  reusability, 
operational  ease,  etc.  Weight  scale:  0  -  not  present,  1  -  minor 
influence,  5  -  strong  influence. 

Another  term  for  failure-directed  testing. 

A  combination  of  top-down  testing  combined  with  bottom-up  testing 
of  prioritized  or  available  components. 

Incremental  analysis  occurs  when  (partial)  analysis  may  be  performed 
on  an  incomplete  product  to  allow  early  feedback  on  the  development 
of  that  product  [YOUN89]. 

Program  statements  sequence  that  can  never  be  executed  [ADRI82]. 
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Inspection 

Instrument 
Integration 
Integration  Testing 

Interface 

Interface  Analysis 
Intrusive  Testing 

lOT&E 

IV&V 

Lifecycle 


A  formal  evaluation  technique  in  which  software  requirements,  design, 
or  code  are  examined  in  detail  by  a  person  or  group  other  than  the 
author  to  detect  faults,  violations  of  development  standards,  and  other 
problems  [IEEE83].  A  quality  improvement  process  for  written 
material  that  consists  of  two  dominant  components:  product 
(document)  improvement  and  process  improvement  (document 
production  and  inspection)  [GILB93]. 

To  install  or  insert  devices  or  instructions  into  hardware  or  software 
to  monitor  the  operation  of  a  system  or  component  [PRIC94]. 

The  process  of  combining  software  components  or  hardware 
components  or  both  into  an  overall  system. 

An  orderly  progression  of  testing  in  which  software  components  or 
hardware  components  or  both  are  combined  and  tested  until  the  entire 
system  has  been  integrated. 

A  shared  boundary.  An  interface  might  be  a  hardware  component  to 
link  two  devices,  or  it  might  be  a  portion  of  storage  or  registers 
accessed  by  two  or  more  computer  programs  [IEEE83]. 

Checks  the  interfaces  between  program  elements  for  consistency  and 
adherence  to  predefined  rules  or  axioms  [YOUN89]. 

Testing  that  collects  timing  and  processing  information  during 
program  execution  that  may  change  the  behavior  of  the  software  from 
its  behavior  in  a  real  environment.  Usually  involves  additional  code 
embedded  in  the  software  being  tested  or  additional  processes  running 
concurrently  with  software  being  tested  on  the  same  platform. 

Initial  Operational  Test  and  Evaluation  is  the  first  phase  of  operational 
test  and  evaluation  conducted  on  pre-protectional  items,  prototypes, 
or  pilot  production  items  and  normally  completed  prior  to  the  first 
major  production  decision.  Conducted  to  provide  a  valid  estimate  of 
expected  system  operational  effectiveness  and  suitability  [YOUN89]. 

Independent  Verification  and  Validation  is  the  verification  and 
validation  of  a  software  product  by  an  organization  that  is  both 
technically  and  managerially  separate  from  the  organization 
responsible  for  developing  the  product  [IEEE83]. 

The  period  that  starts  when  a  software  product  is  conceived  and  ends 
when  the  product  is  no  longer  available  for  use.  The  software 
lifecycle  typically  includes  a  requirements  phase,  design  phase, 
implementation  (code)  phase,  test  phase,  installation  and  checkout 
phase,  operation  and  maintenance  phase,  and  a  retirement  phase 
[IEEE83]. 
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Maintenance 

The  modification  of  a  software  product,  after  delivery,  to  correct 
faults,  to  improve  performance  or  other  attributes,  or  to  adapt  the 
product  to  a  changed  environment. 

Manual  Testing 

That  part  of  software  testing  that  requires  operator  input,  analysis,  or 
evaluation. 

Measure 

To  ascertain  or  appraise  by  comparing  to  a  standard;  to  apply  a  metric 
[IEEE83]. 

Measurement 

(1)  The  act  or  process  of  measuring.  (2)  A  figure,  extent,  or  amount 
obtained  by  measuring  [IEEE83]. 

Metric 

A  measure  of  the  extent  or  degree  to  which  a  product  possesses  and 
exhibits  a  certain  quality,  property,  or  attribute  [IEEE83]. 

Mutation  Testing 

A  method  to  determine  test  set  thoroughness  by  measuring  the  extent 
to  which  a  test  set  can  discriminate  the  program  from  slight  variants 
of  the  program  [ADRI82]. 

Nonintrusive  Testing 

Testing  that  is  transparent  to  the  software  under  test,  i.e.,  testing  that 
does  not  change  the  timing  or  processing  characteristics  of  the 
software  under  test  from  its  behavior  in  a  real  environment.  Usually 
involves  additional  hardware  that  collects  timing  or  processing 
information  and  processes  that  information  on  another  platform. 

Operational  Requirements  Operational  requirements  are  qualitative  and  quantitative  parameters 

that  specify  the  desired  operational  capabilities  of  a  system  and  serve 
as  a  basis  for  determining  the  operational  effectiveness  and  suitability 
of  a  system  prior  to  deployment  [YOUN89]. 

Operational  Testing 

Testing  performed  by  the  end-user  on  software  in  its  normal  operating 
environment  (DoD  usage)  [IEEE83]. 

OT&E 

Operational  Test  and  Evaluation  is  formal  testing  conducted  prior  to 
deployment  to  evaluate  the  operational  effectiveness  and  suitability  of 
the  system  with  respect  to  its  mission  [YOUN89]. 

Outside-ln  Testing 

A  strategy  for  integration  testing  where  units  handling  program  inputs 
and  outputs  are  tested  first,  and  units  that  process  the  inputs  to 
produce  output  being  incrementally  included  as  the  system  is 
integrated  [YOUN89].  A  form  of  hybrid  testing. 

Path  Analysis 

Program  analysis  performed  to  identify  all  possible  paths  through  a 
program,  to  detect  incomplete  paths,  or  to  discover  portions  of  the 
program  that  are  not  on  any  path  [IEEE83]. 

Path  Coverage  Testing 

A  test  method  satisfying  coverage  criteria  that  each  logical  path 
through  the  program  be  tested.  Paths  through  the  program  often  are 
grouped  into  a  finite  set  of  classes;  one  path  from  each  class  is  tested 
[ADRI82]. 

E-202 


Appendix  E:  Glossary 


Peer  Reviews 

Proof  Checker 
Prototyping 

Qualification  Testing 

Quality 

Random  Testing 

Regression  Testing 

Reliability 

Review 


Semantics 


Software  Engineering 
Environment 


Software  Tool 


Peer  reviews  involve  a  methodical  examination  of  software  work 
products  by  the  producer's  peers  to  identify  defects  and  areas  where 
changes  are  needed  [PAUL93]. 

A  program  that  checks  formal  proofs  of  program  properties  for  logical 
correctness  [YOUN89]. 

Prototyping  evaluates  requirements  or  designs  at  the  conceptualization 
phase,  the  requirements  analysis  phase,  or  the  design  phase  by  quickly 
building  scaled-down  components  of  the  intended  system  to  obtain 
rapid  feedback  of  analysis  and  design  decisions. 

Formal  testing,  usually  conducted  by  the  developer  for  the  customer, 
to  demonstrate  that  the  software  meets  its  specified  requirements 
[YOUN89]. 

The  degree  to  which  a  program  possesses  a  desired  combination  of 
attributes  that  enable  it  to  perform  its  specified  end  use  [YOUN89]. 

An  essentially  black-box  testing  approach  in  which  a  program  is  tested 
by  randomly  choosing  a  subset  of  all  possible  input  values.  The 
distribution  may  be  arbitrary  or  may  attempt  to  accurately  reflect  the 
distribution  of  inputs  in  the  application  environment  [YOUN89]. 

Selective  retesting  to  detect  faults  introduced  during  modification  of 
a  system  or  system  component,  to  verify  that  modifications  have  not 
caused  unintended  adverse  effects,  or  to  verify  that  a  modified  system 
or  system  component  still  meets  its  specified  requirements  [IEEE83]. 

The  probability  of  failure-free  operation  for  a  specified  period. 

A  review  is  a  way  to  use  the  diversity  and  power  of  a  group  of  people 
to  point  out  needed  improvements  in  a  product  or  confirm  those  parts 
of  a  product  in  which  improvement  is  either  not  desired  or  not  needed 
[FREE90].  A  review  is  a  general  work  product  evaluation  technique 
that  includes  desk  checking,  walkthroughs,  technical  reviews,  peer 
reviews,  formal  reviews,  and  inspections. 

(1)  The  relationship  of  characters  or  group  of  characters  to  their 
meanings,  independent  of  the  manner  of  their  interpretation  and  use. 

(2)  The  relationships  between  symbols  and  their  meanings  [IEEE83]. 

A  Software  Engineering  Environment  (SEE)  is  a  harmonious 
collection  of  tools  that  provides  automated  support  for  a  software 
engineering  system  that  includes  all  of  the  processes,  methods,  tools, 
information,  and  people  an  organization  uses  to  do  software 
engineering  [SEE94]. 

A  computer  program  used  to  help  develop,  test,  analyze,  or  maintain 
another  computer  program  or  its  documentation;  for  example. 
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Statement  Coverage 
Testing 

Static  Analysis 


Structural  Coverage 


automated  design  tools,  compilers,  test  tools,  and  maintenance  tools 
[IEEE83]. 

A  test  method  satisfying  coverage  criteria  that  requires  each  statement 
be  executed  at  least  once. 

The  process  of  evaluating  a  program  without  executing  the  program 
[IEEE83]. 

Structural  coverage  requires  that  each  pair  of  module  invocations  be 
executed  at  least  once. 


Structural  Testing 
Stub 

Syntax 

System 

System  Simulation 
System  Testing 

Technical  Review 

Test  Bed 

Test  Case 

Test  Development 


A  testing  method  where  the  test  data  are  derived  solely  from  the 
program  structure  [ADRI82]. 

A  software  component  that  usually  minimally  simulates  the  actions  of 
called  components  that  have  not  yet  been  integrated  during  top-down 
testing. 

Syntax  is  (1)  the  relationship  among  characters  or  groups  of 
characters  independent  of  their  meanings  or  the  manner  of  their 
interpretation  and  use,  (2)  the  structure  of  expressions  in  a  language, 
and  (3)  the  rules  governing  the  structure  of  the  language  [IEEE83]. 

A  collection  of  people,  machines,  and  methods  organized  to 
accomplish  a  set  of  specified  functions  [1EEE83]. 

Another  term  for  prototyping. 

The  process  of  testing  an  integrated  hardware  and  software  system  to 
verify  that  the  system  meets  its  specified  requirements  [IEEE83]. 

A  technical  review  is  a  review  that  refers  to  content  of  the  technical 
material  being  reviewed  [FREE90]. 

(1)  An  environment  that  contains  the  integral  hardware, 
instrumentation,  simulators,  software  tools,  and  other  support 
elements  needed  to  conduct  a  test  of  a  logically  or  physically  separate 
component.  (2)  A  suite  of  test  programs  used  in  conducting  the  test 
of  a  component  or  system  [PRIC94]. 

The  definition  of  "test  case"  differs  from  company  to  company, 
engineer  to  engineer,  and  even  project  to  project.  A  test  case  usually 
includes  an  identified  set  of  information  about  observable  states, 
conditions,  events,  and  data  including  inputs  and  expected  outputs. 

The  development  of  anything  required  to  conduct  testing.  This  may 
include  test  requirements  (objectives),  strategies,  processes,  plans, 
software,  procedures,  cases,  documentation,  etc.  [PRIC94]. 
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Test  Driver 

Test  Executive 
Test  Harness 

Test  Plan 

Test  Procedure 

Testing 


Top-Down  Testing 

Unit  Testing 

Validation 

Verification 

Walkthrough 


(1)  A  software  component  that  as  a  rule  minimally  simulates  the 
actions  of  high-level  components  that  have  not  yet  been  integrated 
during  bottom-up  testing.  (2)  Another  term  for  test  harness. 

Another  term  for  test  harness. 

A  software  tool  that  enables  the  testing  of  software  components  that 
links  test  capabilities  to  perform  specific  tests,  accept  program  inputs, 
simulate  missing  components,  compare  actual  outputs  with  expected 
outputs  to  determine  correctness,  and  report  discrepancies  [CLAR91]. 

A  formal  or  informal  plan  to  be  followed  to  assure  the  controlled 
testing  of  the  product  under  test  [PRIC94]. 

The  formal  or  informal  procedure  that  will  be  followed  to  execute  a 
test.  This  is  usually  a  written  document  that  allows  others  to  execute 
the  test  with  a  minimum  of  training  [PRIC94]. 

Any  activity  aimed  at  evaluating  an  attribute  or  capability  of  a 
program  or  system  to  determine  that  it  meets  its  required  results 
[HETZ88].  The  process  of  exercising  or  evaluating  a  system  or 
system  component  by  manual  or  automated  means  to  verify  that  it 
satisfies  specified  requirements  or  to  identify  differences  between 
expected  and  actual  results  [IEEE83]. 

An  integration  testing  technique  that  tests  the  high-level  components 
first  using  stubs  for  lower-level  called  components  that  have  not  yet 
been  integrated  and  that  simulate  the  required  actions  of  those 
components. 

Unit  testing  is  the  testing  that  we  do  to  show  that  a  unit  (the  smallest 
piece  of  software  that  can  be  independently  compiled  or  assembled, 
loaded,  and  tested)  does  not  satisfy  its  functional  specification  or  its 
implemented  structure  does  not  match  the  intended  design  structure 
[BEIZ90]. 

The  process  of  evaluating  software  to  determine  compliance  with 
specified  requirements  [216788]. 

The  process  of  evaluating  the  products  of  a  given  software 
development  activity  to  determine  correctness  and  consistency  with 
respect  to  the  products  and  standards  provided  as  input  to  that  activity 
[216788]. 

In  the  most  usual  form  of  the  term,  a  walkthrough  is  a  step-by-step 
simulation  of  the  execution  of  a  procedure,  as  when  walking  through 
code,  line  by  line,  with  an  imagined  set  of  inputs.  The  term  has  been 
extended  to  the  review  of  material  that  is  not  procedural,  such  as  data 
descriptions,  reference  manuals,  specifications,  etc.  [FREE90]. 
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White-Box  Testing 


Testing  approaches  that  examine  the  program  structure  and  derive  test 
data  from  the  program  logic  [YOUN89]. 
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